{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "from fastai2.vision.all import *\n",
    "from utils import *\n",
    "\n",
    "matplotlib.rc('image', cmap='Greys')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Under the Hood: Training a Digit Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pixels: The Foundations of Computer Vision"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sidebar: Tenacity and Deep Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End sidebar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = untar_data(URLs.MNIST_SAMPLE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "Path.BASE_PATH = path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#9) [Path('cleaned.csv'),Path('item_list.txt'),Path('trained_model.pkl'),Path('models'),Path('valid'),Path('labels.csv'),Path('export.pkl'),Path('history.csv'),Path('train')]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#2) [Path('train/7'),Path('train/3')]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(path/'train').ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#6131) [Path('train/3/10.png'),Path('train/3/10000.png'),Path('train/3/10011.png'),Path('train/3/10031.png'),Path('train/3/10034.png'),Path('train/3/10042.png'),Path('train/3/10052.png'),Path('train/3/1007.png'),Path('train/3/10074.png'),Path('train/3/10091.png')...]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "threes = (path/'train'/'3').ls().sorted()\n",
    "sevens = (path/'train'/'7').ls().sorted()\n",
    "threes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAA9ElEQVR4nM3Or0sDcRjH8c/pgrfBVBjCgibThiKIyTWbWF1bORhGwxARxH/AbtW0JoIGwzXRYhJhtuFY2q1ocLgbe3sGReTuuWbwkx6+r+/zQ/pncX6q+YOldSe6nG3dn8U/rTQ70L8FCGJUewvxl7NTmezNb8xIkvKugr1HSeMP6SrWOVkoTEuSyh0Gm2n3hQyObMnXnxkempRrvgD+gokzwxFAr7U7YXHZ8x4A/Dl7rbu6D2yl3etcw/F3nZgfRVI7rXM7hMUUqzzBec427x26rkmlkzEEa4nnRqnSOH2F0UUx0ePzlbuqMXAHgN6GY9if5xP8dmtHFfwjuQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<PIL.PngImagePlugin.PngImageFile image mode=L size=28x28 at 0x7F10D53E7E90>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "im3_path = threes[1]\n",
    "im3 = Image.open(im3_path)\n",
    "im3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  0,   0,   0,   0,   0,   0],\n",
       "       [  0,   0,   0,   0,   0,  29],\n",
       "       [  0,   0,   0,  48, 166, 224],\n",
       "       [  0,  93, 244, 249, 253, 187],\n",
       "       [  0, 107, 253, 253, 230,  48],\n",
       "       [  0,   3,  20,  20,  15,   0]], dtype=uint8)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "array(im3)[4:10,4:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  0,   0,   0,   0,   0,   0],\n",
       "        [  0,   0,   0,   0,   0,  29],\n",
       "        [  0,   0,   0,  48, 166, 224],\n",
       "        [  0,  93, 244, 249, 253, 187],\n",
       "        [  0, 107, 253, 253, 230,  48],\n",
       "        [  0,   3,  20,  20,  15,   0]], dtype=torch.uint8)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor(im3)[4:10,4:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_5949cb40_7ea5_11ea_8470_251223b34593row0_col0 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row0_col1 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row0_col2 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row0_col3 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row0_col4 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row0_col5 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row0_col6 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row0_col7 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row0_col8 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row0_col9 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row0_col10 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row0_col11 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row0_col12 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row0_col13 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row0_col14 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row0_col15 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row0_col16 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row0_col17 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row1_col0 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row1_col1 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row1_col2 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row1_col3 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row1_col4 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row1_col5 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #efefef;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row1_col6 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #7c7c7c;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row1_col7 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #4a4a4a;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row1_col8 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row1_col9 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row1_col10 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row1_col11 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #606060;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row1_col12 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #4d4d4d;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row1_col13 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #7c7c7c;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row1_col14 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #bbbbbb;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row1_col15 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row1_col16 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row1_col17 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row2_col0 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row2_col1 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row2_col2 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row2_col3 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #e4e4e4;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row2_col4 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #6b6b6b;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row2_col5 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row2_col6 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row2_col7 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row2_col8 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #171717;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row2_col9 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #4b4b4b;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row2_col10 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #010101;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row2_col11 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row2_col12 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row2_col13 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row2_col14 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #171717;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row2_col15 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row2_col16 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row2_col17 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row3_col0 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row3_col1 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #272727;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row3_col2 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #0a0a0a;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row3_col3 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #050505;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row3_col4 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row3_col5 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #333333;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row3_col6 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #e6e6e6;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row3_col7 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #fafafa;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row3_col8 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #fbfbfb;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row3_col9 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #fdfdfd;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row3_col10 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #fafafa;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row3_col11 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #4b4b4b;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row3_col12 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row3_col13 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row3_col14 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #171717;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row3_col15 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row3_col16 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row3_col17 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row4_col0 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row4_col1 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row4_col2 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row4_col3 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row4_col4 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #1b1b1b;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row4_col5 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #e0e0e0;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row4_col6 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row4_col7 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row4_col8 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row4_col9 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row4_col10 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row4_col11 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #4e4e4e;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row4_col12 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row4_col13 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row4_col14 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #767676;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row4_col15 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row4_col16 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row4_col17 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row5_col0 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row5_col1 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #fcfcfc;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row5_col2 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #f6f6f6;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row5_col3 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #f6f6f6;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row5_col4 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #f8f8f8;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row5_col5 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row5_col6 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row5_col7 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row5_col8 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row5_col9 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row5_col10 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #e8e8e8;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row5_col11 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #222222;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row5_col12 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row5_col13 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #090909;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row5_col14 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #d0d0d0;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row5_col15 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row5_col16 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row5_col17 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row6_col0 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row6_col1 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row6_col2 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row6_col3 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row6_col4 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row6_col5 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row6_col6 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row6_col7 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row6_col8 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row6_col9 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row6_col10 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #060606;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row6_col11 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row6_col12 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #090909;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row6_col13 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #979797;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row6_col14 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row6_col15 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row6_col16 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row6_col17 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row7_col0 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row7_col1 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row7_col2 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row7_col3 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row7_col4 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row7_col5 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row7_col6 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row7_col7 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #f8f8f8;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row7_col8 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #b6b6b6;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row7_col9 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #252525;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row7_col10 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #010101;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row7_col11 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #060606;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row7_col12 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #999999;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row7_col13 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row7_col14 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row7_col15 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row7_col16 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row7_col17 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row8_col0 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row8_col1 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row8_col2 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row8_col3 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row8_col4 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row8_col5 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #f9f9f9;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row8_col6 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #6b6b6b;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row8_col7 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #101010;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row8_col8 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #010101;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row8_col9 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #020202;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row8_col10 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #010101;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row8_col11 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #545454;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row8_col12 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #f1f1f1;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row8_col13 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row8_col14 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row8_col15 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row8_col16 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row8_col17 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row9_col0 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row9_col1 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row9_col2 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row9_col3 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row9_col4 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row9_col5 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #f7f7f7;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row9_col6 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #060606;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row9_col7 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #030303;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row9_col8 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #010101;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row9_col9 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #020202;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row9_col10 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #010101;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row9_col11 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row9_col12 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #181818;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row9_col13 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #303030;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row9_col14 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #a9a9a9;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row9_col15 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #fefefe;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row9_col16 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row9_col17 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row10_col0 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row10_col1 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row10_col2 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row10_col3 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row10_col4 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row10_col5 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row10_col6 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row10_col7 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #e8e8e8;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row10_col8 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #bababa;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row10_col9 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #bababa;\n",
       "            color:  #000000;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row10_col10 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #393939;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row10_col11 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row10_col12 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row10_col13 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row10_col14 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row10_col15 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row10_col16 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #000000;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_5949cb40_7ea5_11ea_8470_251223b34593row10_col17 {\n",
       "            font-size:  6pt;\n",
       "            background-color:  #ffffff;\n",
       "            color:  #000000;\n",
       "        }</style><table id=\"T_5949cb40_7ea5_11ea_8470_251223b34593\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >0</th>        <th class=\"col_heading level0 col1\" >1</th>        <th class=\"col_heading level0 col2\" >2</th>        <th class=\"col_heading level0 col3\" >3</th>        <th class=\"col_heading level0 col4\" >4</th>        <th class=\"col_heading level0 col5\" >5</th>        <th class=\"col_heading level0 col6\" >6</th>        <th class=\"col_heading level0 col7\" >7</th>        <th class=\"col_heading level0 col8\" >8</th>        <th class=\"col_heading level0 col9\" >9</th>        <th class=\"col_heading level0 col10\" >10</th>        <th class=\"col_heading level0 col11\" >11</th>        <th class=\"col_heading level0 col12\" >12</th>        <th class=\"col_heading level0 col13\" >13</th>        <th class=\"col_heading level0 col14\" >14</th>        <th class=\"col_heading level0 col15\" >15</th>        <th class=\"col_heading level0 col16\" >16</th>        <th class=\"col_heading level0 col17\" >17</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_5949cb40_7ea5_11ea_8470_251223b34593level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row0_col0\" class=\"data row0 col0\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row0_col1\" class=\"data row0 col1\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row0_col2\" class=\"data row0 col2\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row0_col3\" class=\"data row0 col3\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row0_col4\" class=\"data row0 col4\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row0_col5\" class=\"data row0 col5\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row0_col6\" class=\"data row0 col6\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row0_col7\" class=\"data row0 col7\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row0_col8\" class=\"data row0 col8\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row0_col9\" class=\"data row0 col9\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row0_col10\" class=\"data row0 col10\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row0_col11\" class=\"data row0 col11\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row0_col12\" class=\"data row0 col12\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row0_col13\" class=\"data row0 col13\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row0_col14\" class=\"data row0 col14\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row0_col15\" class=\"data row0 col15\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row0_col16\" class=\"data row0 col16\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row0_col17\" class=\"data row0 col17\" >0</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_5949cb40_7ea5_11ea_8470_251223b34593level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row1_col0\" class=\"data row1 col0\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row1_col1\" class=\"data row1 col1\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row1_col2\" class=\"data row1 col2\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row1_col3\" class=\"data row1 col3\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row1_col4\" class=\"data row1 col4\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row1_col5\" class=\"data row1 col5\" >29</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row1_col6\" class=\"data row1 col6\" >150</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row1_col7\" class=\"data row1 col7\" >195</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row1_col8\" class=\"data row1 col8\" >254</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row1_col9\" class=\"data row1 col9\" >255</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row1_col10\" class=\"data row1 col10\" >254</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row1_col11\" class=\"data row1 col11\" >176</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row1_col12\" class=\"data row1 col12\" >193</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row1_col13\" class=\"data row1 col13\" >150</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row1_col14\" class=\"data row1 col14\" >96</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row1_col15\" class=\"data row1 col15\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row1_col16\" class=\"data row1 col16\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row1_col17\" class=\"data row1 col17\" >0</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_5949cb40_7ea5_11ea_8470_251223b34593level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row2_col0\" class=\"data row2 col0\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row2_col1\" class=\"data row2 col1\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row2_col2\" class=\"data row2 col2\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row2_col3\" class=\"data row2 col3\" >48</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row2_col4\" class=\"data row2 col4\" >166</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row2_col5\" class=\"data row2 col5\" >224</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row2_col6\" class=\"data row2 col6\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row2_col7\" class=\"data row2 col7\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row2_col8\" class=\"data row2 col8\" >234</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row2_col9\" class=\"data row2 col9\" >196</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row2_col10\" class=\"data row2 col10\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row2_col11\" class=\"data row2 col11\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row2_col12\" class=\"data row2 col12\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row2_col13\" class=\"data row2 col13\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row2_col14\" class=\"data row2 col14\" >233</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row2_col15\" class=\"data row2 col15\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row2_col16\" class=\"data row2 col16\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row2_col17\" class=\"data row2 col17\" >0</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_5949cb40_7ea5_11ea_8470_251223b34593level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row3_col0\" class=\"data row3 col0\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row3_col1\" class=\"data row3 col1\" >93</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row3_col2\" class=\"data row3 col2\" >244</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row3_col3\" class=\"data row3 col3\" >249</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row3_col4\" class=\"data row3 col4\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row3_col5\" class=\"data row3 col5\" >187</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row3_col6\" class=\"data row3 col6\" >46</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row3_col7\" class=\"data row3 col7\" >10</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row3_col8\" class=\"data row3 col8\" >8</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row3_col9\" class=\"data row3 col9\" >4</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row3_col10\" class=\"data row3 col10\" >10</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row3_col11\" class=\"data row3 col11\" >194</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row3_col12\" class=\"data row3 col12\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row3_col13\" class=\"data row3 col13\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row3_col14\" class=\"data row3 col14\" >233</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row3_col15\" class=\"data row3 col15\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row3_col16\" class=\"data row3 col16\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row3_col17\" class=\"data row3 col17\" >0</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_5949cb40_7ea5_11ea_8470_251223b34593level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row4_col0\" class=\"data row4 col0\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row4_col1\" class=\"data row4 col1\" >107</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row4_col2\" class=\"data row4 col2\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row4_col3\" class=\"data row4 col3\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row4_col4\" class=\"data row4 col4\" >230</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row4_col5\" class=\"data row4 col5\" >48</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row4_col6\" class=\"data row4 col6\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row4_col7\" class=\"data row4 col7\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row4_col8\" class=\"data row4 col8\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row4_col9\" class=\"data row4 col9\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row4_col10\" class=\"data row4 col10\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row4_col11\" class=\"data row4 col11\" >192</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row4_col12\" class=\"data row4 col12\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row4_col13\" class=\"data row4 col13\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row4_col14\" class=\"data row4 col14\" >156</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row4_col15\" class=\"data row4 col15\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row4_col16\" class=\"data row4 col16\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row4_col17\" class=\"data row4 col17\" >0</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_5949cb40_7ea5_11ea_8470_251223b34593level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row5_col0\" class=\"data row5 col0\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row5_col1\" class=\"data row5 col1\" >3</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row5_col2\" class=\"data row5 col2\" >20</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row5_col3\" class=\"data row5 col3\" >20</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row5_col4\" class=\"data row5 col4\" >15</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row5_col5\" class=\"data row5 col5\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row5_col6\" class=\"data row5 col6\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row5_col7\" class=\"data row5 col7\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row5_col8\" class=\"data row5 col8\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row5_col9\" class=\"data row5 col9\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row5_col10\" class=\"data row5 col10\" >43</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row5_col11\" class=\"data row5 col11\" >224</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row5_col12\" class=\"data row5 col12\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row5_col13\" class=\"data row5 col13\" >245</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row5_col14\" class=\"data row5 col14\" >74</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row5_col15\" class=\"data row5 col15\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row5_col16\" class=\"data row5 col16\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row5_col17\" class=\"data row5 col17\" >0</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_5949cb40_7ea5_11ea_8470_251223b34593level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row6_col0\" class=\"data row6 col0\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row6_col1\" class=\"data row6 col1\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row6_col2\" class=\"data row6 col2\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row6_col3\" class=\"data row6 col3\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row6_col4\" class=\"data row6 col4\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row6_col5\" class=\"data row6 col5\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row6_col6\" class=\"data row6 col6\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row6_col7\" class=\"data row6 col7\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row6_col8\" class=\"data row6 col8\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row6_col9\" class=\"data row6 col9\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row6_col10\" class=\"data row6 col10\" >249</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row6_col11\" class=\"data row6 col11\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row6_col12\" class=\"data row6 col12\" >245</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row6_col13\" class=\"data row6 col13\" >126</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row6_col14\" class=\"data row6 col14\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row6_col15\" class=\"data row6 col15\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row6_col16\" class=\"data row6 col16\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row6_col17\" class=\"data row6 col17\" >0</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_5949cb40_7ea5_11ea_8470_251223b34593level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row7_col0\" class=\"data row7 col0\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row7_col1\" class=\"data row7 col1\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row7_col2\" class=\"data row7 col2\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row7_col3\" class=\"data row7 col3\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row7_col4\" class=\"data row7 col4\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row7_col5\" class=\"data row7 col5\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row7_col6\" class=\"data row7 col6\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row7_col7\" class=\"data row7 col7\" >14</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row7_col8\" class=\"data row7 col8\" >101</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row7_col9\" class=\"data row7 col9\" >223</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row7_col10\" class=\"data row7 col10\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row7_col11\" class=\"data row7 col11\" >248</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row7_col12\" class=\"data row7 col12\" >124</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row7_col13\" class=\"data row7 col13\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row7_col14\" class=\"data row7 col14\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row7_col15\" class=\"data row7 col15\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row7_col16\" class=\"data row7 col16\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row7_col17\" class=\"data row7 col17\" >0</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_5949cb40_7ea5_11ea_8470_251223b34593level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row8_col0\" class=\"data row8 col0\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row8_col1\" class=\"data row8 col1\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row8_col2\" class=\"data row8 col2\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row8_col3\" class=\"data row8 col3\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row8_col4\" class=\"data row8 col4\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row8_col5\" class=\"data row8 col5\" >11</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row8_col6\" class=\"data row8 col6\" >166</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row8_col7\" class=\"data row8 col7\" >239</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row8_col8\" class=\"data row8 col8\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row8_col9\" class=\"data row8 col9\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row8_col10\" class=\"data row8 col10\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row8_col11\" class=\"data row8 col11\" >187</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row8_col12\" class=\"data row8 col12\" >30</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row8_col13\" class=\"data row8 col13\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row8_col14\" class=\"data row8 col14\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row8_col15\" class=\"data row8 col15\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row8_col16\" class=\"data row8 col16\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row8_col17\" class=\"data row8 col17\" >0</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_5949cb40_7ea5_11ea_8470_251223b34593level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row9_col0\" class=\"data row9 col0\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row9_col1\" class=\"data row9 col1\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row9_col2\" class=\"data row9 col2\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row9_col3\" class=\"data row9 col3\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row9_col4\" class=\"data row9 col4\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row9_col5\" class=\"data row9 col5\" >16</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row9_col6\" class=\"data row9 col6\" >248</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row9_col7\" class=\"data row9 col7\" >250</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row9_col8\" class=\"data row9 col8\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row9_col9\" class=\"data row9 col9\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row9_col10\" class=\"data row9 col10\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row9_col11\" class=\"data row9 col11\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row9_col12\" class=\"data row9 col12\" >232</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row9_col13\" class=\"data row9 col13\" >213</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row9_col14\" class=\"data row9 col14\" >111</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row9_col15\" class=\"data row9 col15\" >2</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row9_col16\" class=\"data row9 col16\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row9_col17\" class=\"data row9 col17\" >0</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_5949cb40_7ea5_11ea_8470_251223b34593level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row10_col0\" class=\"data row10 col0\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row10_col1\" class=\"data row10 col1\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row10_col2\" class=\"data row10 col2\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row10_col3\" class=\"data row10 col3\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row10_col4\" class=\"data row10 col4\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row10_col5\" class=\"data row10 col5\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row10_col6\" class=\"data row10 col6\" >0</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row10_col7\" class=\"data row10 col7\" >43</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row10_col8\" class=\"data row10 col8\" >98</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row10_col9\" class=\"data row10 col9\" >98</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row10_col10\" class=\"data row10 col10\" >208</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row10_col11\" class=\"data row10 col11\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row10_col12\" class=\"data row10 col12\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row10_col13\" class=\"data row10 col13\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row10_col14\" class=\"data row10 col14\" >253</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row10_col15\" class=\"data row10 col15\" >187</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row10_col16\" class=\"data row10 col16\" >22</td>\n",
       "                        <td id=\"T_5949cb40_7ea5_11ea_8470_251223b34593row10_col17\" class=\"data row10 col17\" >0</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f10d4f3b8d0>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "im3_t = tensor(im3)\n",
    "df = pd.DataFrame(im3_t[4:15,4:22])\n",
    "df.style.set_properties(**{'font-size':'6pt'}).background_gradient('Greys')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First Try: Pixel Similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6131, 6265)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seven_tensors = [tensor(Image.open(o)) for o in sevens]\n",
    "three_tensors = [tensor(Image.open(o)) for o in threes]\n",
    "len(three_tensors),len(seven_tensors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAEQAAABECAYAAAA4E5OyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAADjElEQVR4nO2aPyh9YRjHP/f4k38L5X+ysohsUpTBhEVMJGUyGAwWg0kGkcFqlMFIyv+kSGIwKWUiUvKn5P/9DXrvcR+He+695957+vV8llPnvvd9n77n2/s8z3tOIBgMothYqQ7Ab6ggAhVEoIIIVBBBeoTf/+cUFHC6qQ4RqCACFUSggghUEIEKIlBBBCqIQAURRKpUPeHh4QGAyclJAI6PjwFYXl4GIBgMEgh8FY59fX0A3N7eAlBTUwNAU1MTAC0tLQmNVR0iCEQ4MYupl7m4uABgYmICgJWVFQDOz8/DxhUVFQFQX18fGvMbxcXFAFxeXsYSkhPay7jBkz1ke3sbgLa2NgBeX18BeH9/B6CzsxOAnZ0dAAoLCwFC+4ZlWXx8fISNXVpa8iK0qFGHCDxxyN3dHQBPT09h98vLywGYmpoCoKys7Nc5LMsKu0p6enrijtMN6hCBJ1nm8/MTgOfn57D75mlnZWVFnOPq6gqAxsZGwM5I2dnZAOzu7gJQW1vrJiQ3aJZxgyd7iHFCTk5OzHNUVlYCdmYyzjDVrYfO+BN1iCApvYzk5eUFgM3NTQCGhoZCzsjMzARgenoagIGBgaTGpg4RJMUhpnIdHh4GYH5+HrDrl++0t7cD0NXVlYzQfqAOESSk25WY+iQ/Px8g1LeYqxMlJSUAlJaWAjAyMgLYvY7pg+LAcYKkCCIxRdjJyUno3tjYGAD7+/t//tcIMjc3B0Bubm6sYWhh5oaUOMSJt7c3wHaPScn9/f2O4w8PDwGoq6uLdUl1iBtSUpg5kZGRAUBFRQUAvb29AKyurgKwsLAQNn5tbQ2IyyGOqEMEvnGIxKTV39JrdXV1QtZVhwh8k2Uke3t7ADQ3NwP2sYDh5uYGgIKCgliX0CzjBt/tIWdnZwAMDg4CP51h6pK8vLyErK8OEfhmDzF1RUdHB2AfIhnMEePp6Slg1y1xoHuIG1K6h1xfXwMwOzvL+Pg48PVpxHfMS+6trS3AE2f8iTpE4KlDzBPf2NgA7I9bHh8fATg4OADg6OgIsM807u/vQ3OkpaUB9qvLmZkZIHFZRaIOEXiaZbq7uwFYXFyMOpDW1lYARkdHAWhoaIh6jijRLOMGTx1iPnIxtUQkzEHy+vo6VVVVXwHFf3jsFnWIG3xTqaYAdYgbVBCBCiJQQQQqiCBSL5O0osAvqEMEKohABRGoIAIVRKCCCP4B/PMI7HrW9/wAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 72x72 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_image(three_tensors[1]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([6131, 28, 28])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stacked_sevens = torch.stack(seven_tensors).float()/255\n",
    "stacked_threes = torch.stack(three_tensors).float()/255\n",
    "stacked_threes.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(stacked_threes.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stacked_threes.ndim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAEQAAABECAYAAAA4E5OyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAE1klEQVR4nO2byU8jPRTEf2En7AgQO4gDi9hO8P9fOIE4AGIR+xoU1kAgQAKZA6o4eUMU6O7R983IdbE66XYHv3K98rOJ5fN5PByq/usf8H+DHxADPyAGfkAM/IAY1FT4/l9OQbGvPvQMMfADYuAHxMAPiIEfEINKWSYSaL1kW/t9MWKx2JfX5T6PCp4hBpEwxEb+/f0dgFwuB0A2mwXg6emppH1+fgbg9fWVj48PAGprawGIx+MANDc3A9DU1ARAfX19yX3V1dVAeQb9FJ4hBqEYIkZYJmQyGQBub28BuLi4AGBnZweAvb09AM7PzwFIJpOFPsSEvr4+AMbHxwGYnZ0FYHR0FICuri7AMUiMqar6jHFQpniGGARiSDmtkDZcXV0BsL+/D8Da2hoAu7u7ABwcHACOOalUipeXl5J3tLW1AXB6elrS58LCAgBTU1MA9Pf3A44pYkhQeIYYhGKIMoMYoigXZw+AmprP13R2dgJuvo+NjQGf2qNnbm5uSvp4e3sD4O7uDnC6I41Rn8pK+m1eQyJCJFlG0VDkW1tbARgaGgKgo6MDcFEX5CFyuVwhIx0dHQFwcnICOF3SO8RGsbOc+w0KzxCDUAyRoivSmse6lvIrGymqgqKayWRIJBIAXF9fl/Ste6RD8il6V11dXcn93qlGjEAMURQUFUVP19ISO8/FFGWfx8dH4NNjbGxsALC1tQU4DWloaAAc2wYGBgCXXRobGwHHyrDwDDGIREMsY8QMtVrjKMtcXl4CsLm5CcDq6irr6+uAc7fqc35+HoDe3l7AOVNlsqjWMIW/KdTT/yBCaUglVyjNSKVSgIv+0tISACsrKwAsLy8XHKhYJScqBrS0tABOU6JmhuAZYhBKQyxTBF0rm2ilKp3Q6lcMSSQSBWbIX6gyJv2RPxHburu7S+6zlbOgiLTIbBd9tjygHytBnJ6eBmBkZKTQh50KelZlABWZ2tvbATeFoiol+iljEMnizk4Zu9iTiVIZUNcyZvl8vsAmlR+TySRAwdI/PDwAcHh4CMDw8DDgmGItfFCj5hliEKpAZDWjXDnAFnGkGcXPSyskmipEizlKy/peDBJTrFELWijyDDH4EUMsI2xrmaPoKDVqnlsUM0T3SDO03NcC0pYrldpticFrSEQIpCG2uKxWURLEEEVLGcBuFcRisd88ixiirKN32ixiWesXdxEjlIbYTWw7nxUt6YLdqC4uHIsRcqTb29uA8yF6l5yptEV9e6f6hxDKh2i+q/CjzSQ5UGUCRdEu3IR0Os3x8THgmCEfoj61ua1FXU9PD+BKi9apBoVniMGPGGLnp90qSKfTgCsEyV1KY3SfXcmmUqlCiUDPaAtzcHAQcI50cnIScAUkOVT5FJ9lIkYgDZGiKyrSBm0JCPf394DTA2UQfS6NyWazBdboGMTMzAwAc3NzACwuLgIwMTEBOA0pVw8JCs8Qg0AaomhK2VUA1haBXcvY+8/OzgDnQuPxeEEjdERCtZNymmFLh2Gzi+AZYhCrcIzgyy8rHcOUY1V2kQtVK99SfBRTkbetdMkew4xg+8H/e8h3EIghZW8uc4S70tFu+N3jVGojgGfIdxApQ/4yeIZ8B35ADPyAGFRyqtH+d85fAM8QAz8gBn5ADPyAGPgBMfADYvALMumtb+Vr5kIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 72x72 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mean3 = stacked_threes.mean(0)\n",
    "show_image(mean3);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAEQAAABECAYAAAA4E5OyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAElUlEQVR4nO2bSUszWRSGn3JKUGOM84giCA7oQnHj33ejiKCI4MIpxhES5ylx6oW8dTvnMyaWJd1033dzqVRqyLlPnelWgvf3d7yc6v7pG/i3yRvEyBvEyBvEyBvEqKHK/v9yCAo++9ATYuQNYuQNYuQNYuQNYuQNYuQNYuQNYuQNYlQtU42kaj2Wz/YHwaeJY+TvRZUnxOhHhGimNb69vZWNr6+vX25r/Ep1dR9zVl9f/3HDDQ1l29qvUQRFJckTYvQtQiwRmvGXlxcAnp6eALi9vQXg+voagMvLSwAuLi7KxoeHh/A4nUOjrtHU1ARAR0cHAENDQwAMDw8D0NvbC0BraysAjY2NgCPou6R4QowiEaJZfH5+BhwR+XwegIODAwC2t7cB2NvbA+Dw8BCAo6MjAK6urgAoFovhuUSdRhEiMubn5wFYXFwEYGFhoWx/JZ9SqzwhRpEIUXTQrN7f3wNQKBQA2N/fB2B3dxdwpIicm5ubsvMlEgkSiQTgyBB1Oufj4yPgfMXo6GjZtXWc5KNMTKqJkGqZp2ZDnr25uRmA9vZ2AEZGRgDo7u4GnF/Q/nQ6HRKiGRdNq6urgPM3IsFe0+YlUeUJMaqJEJv9aRaUNYqIzs5OAMbGxsr29/f3A44Mbff19QEffkEzrBxlaWkJgLOzM8DlOJlMpmxsa2sDXP4RNbpInhCjb0WZaoTYGkVEaDuVSgGOJM1uQ0NDmNtYKdroXD09PYDzS+l0GnCE/LQa9oQYRap2K1WglhRFDn2/paUF+LPuCIIgPEY5irLb8/NzwNUyqmEGBgbKrql7+akiPTKSNYx+oH64DCKDCXttS6+vr2G43dzcBGB9fR1wj8zU1BTgHhWFbHsuSamCT91/qB81iKyTFSkiwTo6jXo8NIulUolsNgvA8vIyADs7O4CjTKFcox6VuFuKnhCjSIRU8iX2ubUNJdtYUnGYz+dZWVkBYG1tDXCJ2OzsLACTk5OAC7vJZLLs2nGR4gkximUZwhZalZrOkvarpM/lcmxsbAAuzKoQVENoZmYGcOFXfsq2Cn1iFrNiiTJ22/oSjXYZQknY1tYWuVwOcDM/NzcHOB8yODgIuKRO+YdtGVa6t1rlCTGK1YdUyg7tfvmO09NTALLZbLgkoTxjenoagImJCcBlprbMj4sMyRNiFOtityVBks8olUqAawIpGy0UCiEBKt7Gx8cB6OrqAqrnHT4P+SXFSkiljFRkaGnz5OQEcO3BIAjCdqKWF7TwpMrZRpXfei3CE2IUCyGVXouwC1la6jw+PgZcvZJKpcJ2oho/2lZe8tPlhVrlCTH6FR+ihrF8h7peWmy6u7sDXE6RyWTCaKJqVv0O+Y64apVq8oQY/corVYou8hHKTLUtMuQnEolE+OLL3z+D6C++RJUnxCiWarfaYriI0EKVljIVhZLJZLjgpIxVmWmlrvpvyRNiFFSZ3Zr+YmZ9iH3lStFGPqRYLJYdFwRBSJHIkA+xL9HF2CHzfzGrRbEQ8sdBFc5po9JX37cE/EIe4gmpRdUI+d/JE2LkDWLkDWLkDWLkDWLkDWL0F7hnDWZImx+vAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 72x72 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mean7 = stacked_sevens.mean(0)\n",
    "show_image(mean7);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAEQAAABECAYAAAA4E5OyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAADjElEQVR4nO2aPyh9YRjHP/f4k38L5X+ysohsUpTBhEVMJGUyGAwWg0kGkcFqlMFIyv+kSGIwKWUiUvKn5P/9DXrvcR+He+695957+vV8llPnvvd9n77n2/s8z3tOIBgMothYqQ7Ab6ggAhVEoIIIVBBBeoTf/+cUFHC6qQ4RqCACFUSggghUEIEKIlBBBCqIQAURRKpUPeHh4QGAyclJAI6PjwFYXl4GIBgMEgh8FY59fX0A3N7eAlBTUwNAU1MTAC0tLQmNVR0iCEQ4MYupl7m4uABgYmICgJWVFQDOz8/DxhUVFQFQX18fGvMbxcXFAFxeXsYSkhPay7jBkz1ke3sbgLa2NgBeX18BeH9/B6CzsxOAnZ0dAAoLCwFC+4ZlWXx8fISNXVpa8iK0qFGHCDxxyN3dHQBPT09h98vLywGYmpoCoKys7Nc5LMsKu0p6enrijtMN6hCBJ1nm8/MTgOfn57D75mlnZWVFnOPq6gqAxsZGwM5I2dnZAOzu7gJQW1vrJiQ3aJZxgyd7iHFCTk5OzHNUVlYCdmYyzjDVrYfO+BN1iCApvYzk5eUFgM3NTQCGhoZCzsjMzARgenoagIGBgaTGpg4RJMUhpnIdHh4GYH5+HrDrl++0t7cD0NXVlYzQfqAOESSk25WY+iQ/Px8g1LeYqxMlJSUAlJaWAjAyMgLYvY7pg+LAcYKkCCIxRdjJyUno3tjYGAD7+/t//tcIMjc3B0Bubm6sYWhh5oaUOMSJt7c3wHaPScn9/f2O4w8PDwGoq6uLdUl1iBtSUpg5kZGRAUBFRQUAvb29AKyurgKwsLAQNn5tbQ2IyyGOqEMEvnGIxKTV39JrdXV1QtZVhwh8k2Uke3t7ADQ3NwP2sYDh5uYGgIKCgliX0CzjBt/tIWdnZwAMDg4CP51h6pK8vLyErK8OEfhmDzF1RUdHB2AfIhnMEePp6Slg1y1xoHuIG1K6h1xfXwMwOzvL+Pg48PVpxHfMS+6trS3AE2f8iTpE4KlDzBPf2NgA7I9bHh8fATg4OADg6OgIsM807u/vQ3OkpaUB9qvLmZkZIHFZRaIOEXiaZbq7uwFYXFyMOpDW1lYARkdHAWhoaIh6jijRLOMGTx1iPnIxtUQkzEHy+vo6VVVVXwHFf3jsFnWIG3xTqaYAdYgbVBCBCiJQQQQqiCBSL5O0osAvqEMEKohABRGoIAIVRKCCCP4B/PMI7HrW9/wAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 72x72 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "a_3 = stacked_threes[1]\n",
    "show_image(a_3);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.1114), tensor(0.2021))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dist_3_abs = (a_3 - mean3).abs().mean()\n",
    "dist_3_sqr = ((a_3 - mean3)**2).mean().sqrt()\n",
    "dist_3_abs,dist_3_sqr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.1586), tensor(0.3021))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dist_7_abs = (a_3 - mean7).abs().mean()\n",
    "dist_7_sqr = ((a_3 - mean7)**2).mean().sqrt()\n",
    "dist_7_abs,dist_7_sqr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.1586), tensor(0.3021))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "F.l1_loss(a_3.float(),mean7), F.mse_loss(a_3,mean7).sqrt()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NumPy Arrays and PyTorch Tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [[1,2,3],[4,5,6]]\n",
    "arr = array (data)\n",
    "tns = tensor(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2, 3],\n",
       "       [4, 5, 6]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr  # numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1, 2, 3],\n",
       "        [4, 5, 6]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tns  # pytorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([4, 5, 6])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tns[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2, 5])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tns[:,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([5, 6])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tns[1,1:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[2, 3, 4],\n",
       "        [5, 6, 7]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tns+1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'torch.LongTensor'"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tns.type()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.5000, 3.0000, 4.5000],\n",
       "        [6.0000, 7.5000, 9.0000]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tns*1.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing Metrics Using Broadcasting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([1010, 28, 28]), torch.Size([1028, 28, 28]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid_3_tens = torch.stack([tensor(Image.open(o)) \n",
    "                            for o in (path/'valid'/'3').ls()])\n",
    "valid_3_tens = valid_3_tens.float()/255\n",
    "valid_7_tens = torch.stack([tensor(Image.open(o)) \n",
    "                            for o in (path/'valid'/'7').ls()])\n",
    "valid_7_tens = valid_7_tens.float()/255\n",
    "valid_3_tens.shape,valid_7_tens.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.1114)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def mnist_distance(a,b): return (a-b).abs().mean((-1,-2))\n",
    "mnist_distance(a_3, mean3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([0.1050, 0.1526, 0.1186,  ..., 0.1122, 0.1170, 0.1086]),\n",
       " torch.Size([1010]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid_3_dist = mnist_distance(valid_3_tens, mean3)\n",
    "valid_3_dist, valid_3_dist.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2, 3, 4])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor([1,2,3]) + tensor([1,1,1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1010, 28, 28])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(valid_3_tens-mean3).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def is_3(x): return mnist_distance(x,mean3) < mnist_distance(x,mean7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(True), tensor(1.))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "is_3(a_3), is_3(a_3).float()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([True, True, True,  ..., True, True, True])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "is_3(valid_3_tens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.9168), tensor(0.9854), tensor(0.9511))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_3s =      is_3(valid_3_tens).float() .mean()\n",
    "accuracy_7s = (1 - is_3(valid_7_tens).float()).mean()\n",
    "\n",
    "accuracy_3s,accuracy_7s,(accuracy_3s+accuracy_7s)/2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stochastic Gradient Descent (SGD)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: G Pages: 1 -->\n",
       "<svg width=\"591pt\" height=\"78pt\"\n",
       " viewBox=\"0.00 0.00 591.49 78.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 74)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-74 587.4867,-74 587.4867,4 -4,4\"/>\n",
       "<!-- init -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>init</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"27\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">init</text>\n",
       "</g>\n",
       "<!-- predict -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>predict</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"126.0969\" cy=\"-18\" rx=\"35.194\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"126.0969\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">predict</text>\n",
       "</g>\n",
       "<!-- init&#45;&gt;predict -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>init&#45;&gt;predict</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M54.0787,-18C62.3227,-18 71.6196,-18 80.7269,-18\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"80.8626,-21.5001 90.8626,-18 80.8625,-14.5001 80.8626,-21.5001\"/>\n",
       "</g>\n",
       "<!-- loss -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>loss</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"225.1938\" cy=\"-52\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"225.1938\" y=\"-48.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">loss</text>\n",
       "</g>\n",
       "<!-- predict&#45;&gt;loss -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>predict&#45;&gt;loss</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M155.2932,-28.0172C166.6224,-31.9043 179.6698,-36.3808 191.4018,-40.406\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"190.2859,-43.7234 200.8806,-43.6582 192.5577,-37.1023 190.2859,-43.7234\"/>\n",
       "</g>\n",
       "<!-- gradient -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>gradient</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"361.8403\" cy=\"-52\" rx=\"39.7935\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"361.8403\" y=\"-48.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">gradient</text>\n",
       "</g>\n",
       "<!-- loss&#45;&gt;gradient -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>loss&#45;&gt;gradient</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M252.5178,-52C269.4967,-52 291.836,-52 311.8929,-52\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"312.1329,-55.5001 322.1329,-52 312.1328,-48.5001 312.1329,-55.5001\"/>\n",
       "</g>\n",
       "<!-- step -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>step</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"465.4867\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"465.4867\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">step</text>\n",
       "</g>\n",
       "<!-- gradient&#45;&gt;step -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>gradient&#45;&gt;step</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M394.0665,-41.4286C405.9515,-37.5298 419.4492,-33.1021 431.4862,-29.1535\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"432.7754,-32.4142 441.1862,-25.9715 430.5935,-25.7629 432.7754,-32.4142\"/>\n",
       "</g>\n",
       "<!-- step&#45;&gt;predict -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>step&#45;&gt;predict</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M438.4132,-18C380.3272,-18 243.2155,-18 171.5401,-18\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"171.4571,-14.5001 161.4571,-18 171.4571,-21.5001 171.4571,-14.5001\"/>\n",
       "<text text-anchor=\"middle\" x=\"287.1938\" y=\"-21.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">repeat</text>\n",
       "</g>\n",
       "<!-- stop -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>stop</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"556.4867\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"556.4867\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">stop</text>\n",
       "</g>\n",
       "<!-- step&#45;&gt;stop -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>step&#45;&gt;stop</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M492.7897,-18C501.068,-18 510.3085,-18 519.1272,-18\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"519.203,-21.5001 529.203,-18 519.203,-14.5001 519.203,-21.5001\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x7f10d475e550>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gv('''\n",
    "init->predict->loss->gradient->step->stop\n",
    "step->predict[label=repeat]\n",
    "''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(x): return x**2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEMCAYAAADeYiHoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3yV5f3/8dcnm0xWElYGYQ9ZBmQ5aVW0ai1UQdwojmptrbVLq1XbX7Xj2yUqiiI46ihK3a1VVBCRsIQgG0ICJCSMkD0/vz/OoY3xBE5Czn2fJJ/n43E/esaVc7+9m5wP133d93WJqmKMMcY0FuJ2AGOMMcHJCoQxxhifrEAYY4zxyQqEMcYYn6xAGGOM8SnM7QCtpXv37pqenu52DGOMaVNWr15dpKqJvt5rNwUiPT2drKwst2MYY0ybIiI5Tb1np5iMMcb4ZAXCGGOMT1YgjDHG+GQFwhhjjE9WIIwxxvjkeIEQkQEiUikizzXxvojIwyJy0Ls9IiLidE5jjOno3LjM9VFg1XHenwN8GxgJKPBvYCfweOCjGWOMOcbRHoSIzACOAP85TrNrgD+oap6q7gX+AFwbqEzrco/w8LubA/XxxhgTMKrKr9/aRPa+4oB8vmMFQkTigQeAH52g6TBgfYPn672v+frMOSKSJSJZhYWFLcq1Ie8Ijy3dwca9gTnAxhgTKJ/tPMSTn+xiS35JQD7fyR7Eg8B8Vc09QbtYoOG3dTEQ62scQlXnqWqmqmYmJvq8U/yELh7Vm8iwEP6+ak+Lft4YY9zy0qo9xEWFMXV4z4B8viMFQkRGAd8A/s+P5qVAfIPn8UCpBmjpu4RO4VxwSk+WrNtHRXVdIHZhjDGtrri8hnc25nPJqF50iggNyD6c6kGcBaQDe0QkH7gLmCYia3y0zcYzQH3MSO9rAXNZZgollbW8s3F/IHdjjDGtZsn6vVTV1jNjbGrA9uFUgZgH9ANGebfHgbeA83y0XQjcKSK9RaQXnjGLBYEMNz6jK+ndonlp1YnOfhljTHB4aVUuQ3vGM7x3QsD24UiBUNVyVc0/tuE5jVSpqoUicrqIlDZo/gTwBrAB2IinkDwRyHwiwnczU1i56xA7C0tP/APGGOOijXuLyd53lBnjUgK6H1fupFbV+1X1Su/jT1Q1tsF7qqp3q2pX73Z3oMYfGpp+ah9CQ4SXs/ICvStjjDkpf1+1h8iwEC4Z2Tug+7GpNryS46M4e1Air67Oo6au3u04xhjjU0V1HUvW7WPq8B4kRIcHdF9WIBq4fGwqRaVVfLD5gNtRjDHGp7c37KekspbLxgb29BJYgfiKswclkhwfyYuf2z0Rxpjg9OLne+jbPYYJGd0Cvi8rEA2EhYZwWWYKH20tZO+RCrfjGGPMV2wrKCEr5zAzxqbgxBymViAauSzT022zS16NMcHmxc9zCQ8Vpp3ax5H9WYFoJKVrNGcMSOSVrFxqbbDaGBMkKmvqWLw2j3OH9aB7bKQj+7QC4cPMcSnsL67ko60tmwDQGGNa23vZ+Rwpr2FmAO+cbswKhA9ThiTTPTaSFz+300zGmODw4ud7SO0azcR+gR+cPsYKhA/hoSFcltmHDzYXkF9c6XYcY0wHt7OwlM92HmLGuBRCQpxbYNMKRBNmjE2lXm2w2hjjvhc/30NYiDDdocHpY6xANCG1WzRnDEzk76v22GC1McY1lTV1vLI6j/OG9SApLsrRfVuBOI5Zp6Wyv7iSD7fYYLUxxh3vbNzPkfIarjjNucHpY6xAHMeUwUkkx0fy/Moct6MYYzqo5z9z7s7pxqxAHEdYaAiXj03lo62F5B4qdzuOMaaD2Zx/lKycw1wxLtXRweljrECcwIyxKQjY/EzGGMe9sHIPEaEhjt053ZhjBUJEnhOR/SJyVES2isgNTbS7VkTqRKS0wXaWUzkb69W5E+cMTublrDyqa22w2hjjjPLqWl5bs5cLTulB15gIVzI42YP4f0C6qsYDFwMPicipTbRdoaqxDbaljqX0YdZ4zzTg72XnuxnDGNOB/HPdPkqqarnitDTXMjhWIFQ1W1Wrjj31bv2c2v/JOGNAIildO/HcZzZYbYwJPFVl4YocBiXHMTa9i2s5HB2DEJG5IlIObAb2A2830XS0iBR5T0XdKyJhTXzeHBHJEpGswsLAXYoaGiLMOi2NlbsOsbWgJGD7McYYgLW5R9i0/yhXTkhzZFrvpjhaIFT1ViAOOB1YDFT5aPYxMBxIAqYBM4EfN/F581Q1U1UzExMTAxPa67LMFCLCQqwXYYwJuOdW5BAbGcalowO75vSJOH4Vk6rWqeoyoA9wi4/3d6rqLlWtV9UNwAPAdKdzNtY1JoJvndKTxWv2UlpV63YcY0w7daismje/2M93xvQmNtLnyRPHuHmZaxj+jUEo4F4fq4ErJ6RRWlXL62v3uh3FGNNOvZyVS3VdPVeOd29w+hhHCoSIJInIDBGJFZFQETkPz6mjD3y0nSoiyd7Hg4F7gSVO5DyR0SmdGdYrnuc+y0FV3Y5jjGln6uqV51fmcFrfrgxMjnM7jmM9CMVzOikPOAz8HviBqi4RkVTvvQ7HJhqZAnwhImV4BrEXA79xKOdxiQhXjU9jc75nXVhjjGlNH28tJPdQBVdNcL/3AJ7TPAGnqoXAmU28tweIbfD8LuAuJ3K1xMWjevGbt7/k2U93Mza9q9txjDHtyLMrdpMYF8m5Q3u4HQWwqTaaLToijMsyU3h3Yz4FR20xIWNM69hdVMbSLYXMOi2ViLDg+GoOjhRtzFUT0qhT5fmVNj+TMaZ1LFyRQ3iouDKtd1OsQLRAWrcYzh6UxAsr99j8TMaYk1ZWVcsrq3OZOryn44sCHY8ViBa6ekIaRaVVvLNxv9tRjDFt3Gtr91JSWcs1E4NjcPoYKxAtdMaARPp2j+HZT3e7HcUY04Z55l3azfDe8YxJdW/eJV+sQLRQSIjnktc1e46wIa/Y7TjGmDZqxc6DbC0o5eoJ6a7Ou+SLFYiTMD2zD9ERoTzz6S63oxhj2qgFy3fTJTqci0f2cjvK11iBOAnxUeFMP7UPb67fT2GJr3kHjTGmabmHynn/ywJmjkslKjzU7ThfYwXiJF0zMZ3qunpesEtejTHNtHDFbs8MDUFy53RjViBOUr/EWM4alMhzK3PskldjjN/Kqmr5+6pcpg7vQc+ETm7H8ckKRCu4dmI6hSVVvLVhn9tRjDFtxOI1eZRU1nLdpHS3ozTJCkQrOGNAIhmJMTyzfLfN8mqMOaH6euWZT3czok9C0F3a2pAViFYQEiJcNzGdL/KKWbPHZnk1xhzfJ9uL2FlYxnWTgu/S1oasQLSS74zpQ3xUGE8v3+12FGNMkHt62S4S4yK58JTgu7S1ISsQrSQmMoyZ41J5Z8N+cg+Vux3HGBOkthWU8NHWQq4enxY0s7Y2xbF0IvKciOwXkaMislVEbjhO2x+KSL6IFIvI0yIS6VTOk3HNRE930abfMMY05enlu4gMC2FWECwpeiJOlq//B6SrajxwMfCQiJzauJF3OdKf4llZLh3IAH7lYM4W69W5Exec0pOXVuVSUlnjdhxjTJA5WFrFP9bs5Ttj+tA1JsLtOCfkWIFQ1WxVPXa7sXq3fj6aXgPM97Y/DDwIXOtMypM3e3JfSqpqeTkrz+0oxpgg87x3iYDZk9PdjuIXR0+AichcESkHNgP78aw53dgwYH2D5+uBZBHp5uPz5ohIlohkFRYWBiRzc41K6UxmWheeWb6Lunq75NUY41FVW8fCFTmcNSiR/klxbsfxi6MFQlVvBeKA04HFgK8JjGKBhtOjHnv8tSOqqvNUNVNVMxMTE1s7bovNntyXvMMV/Cs73+0oxpgg8c91+ygqrWL25L5uR/Gb40PoqlqnqsuAPsAtPpqUAvENnh97XBLobK3l3GE9SOnaiaeW2SyvxhjPmg/zl+1iUHIck/t3dzuO39y8xioM32MQ2cDIBs9HAgWqetCRVK0gNESYPakvq3MOszrnkNtxjDEu+3hbEZvzS7jxjIygvjGuMUcKhIgkicgMEYkVkVDvlUozgQ98NF8IzBaRoSLSBbgHWOBEztb03cwUEjqFM+/jnW5HMca47MmPd5IcHxmUaz4cj1M9CMVzOikPOAz8HviBqi4RkVQRKRWRVABVfRd4BPgQyPFu9zmUs9XERIZx1fg0/rWpgF1FZW7HMca4JHtfMcu2F3HdpL5Bf2NcY46kVdVCVT1TVTuraryqnqKqT3rf26Oqsaq6p0H7P6pqsrftdQ0uj21Trp6YRnhICPOXWS/CmI7qyY93EhMRysxxqW5Haba2Vc7amKS4KC4d3ZtXsvI4WNoma5wx5iTsO1LBG1/sZ8a4VBI6hbsdp9msQATYDaf3paq2nkWf5bgdxRjjsGeWe65kDOY1H47HCkSADUiOY8rgJBauyKGius7tOMYYhxSX1/DCyj18a0RP+nSJdjtOi1iBcMDNZ/XjUFk1L2fluh3FGOOQ51bmUFZdx01n+Lqav22wAuGAseldOTWtC09+spPaOlu32pj2rrKmjmeW7+KMgYkM7RV/4h8IUlYgHHLzmf3IO1zBWxv2ux3FGBNgr67Oo6i0mpvPzHA7ykmxAuGQKYOTGJAUy+Mf7bR1q41px+rqlSc/2cnIPglMyPjaHKNtihUIh4SECHPOyODL/Uf5aGtwzDxrjGl972zcT87Bcm4+s1+bmlbDFysQDrpkVG96JkTx2NIdbkcxxgSAqvL4Rzvo2z2Gc4f1cDvOSbMC4aCIsBBuOD2DlbsOsTrnsNtxjDGt7ONtRWzce5Sbz8wgNKRt9x7ACoTjZo5LoUt0OI8t3e52FGNMK5v74XZ6JkRx6eg+bkdpFVYgHBYdEcb1k/ry/pcH+HL/UbfjGGNaSdbuQ6zcdYgbT89oc5PyNaV9/Fe0MVdPSCc2MszGIoxpR+Yu3UHXmAhmjEtxO0qrsQLhgoTocGaNT+XNL/ax26YCN6bNy95XzAebD3DdxHSiI8LcjtNqnFowKFJE5otIjoiUiMhaEZnaRNtrRaTOu0bEse0sJ3I6afbkvoSFhvD4R9aLMKate2zpDmIjw7h6QrrbUVqVUz2IMCAXOBNIAO4FXhaR9Cbar/CuEXFsW+pISgclxUUxY2wK/1iTx94jFW7HMca00PYDpby1YT9Xjk8jIbrtTel9PE4tGFSmqver6m5VrVfVN4FdwKlO7D9Y3XSmZxKvJ6wXYUybNffD7USGhXDD6X3djtLqXBmDEJFkYCCQ3UST0SJSJCJbReReEfF5Uk9E5ohIlohkFRa2vbuTe3fuxLQxffj7qlwOHK10O44xpplyDpaxZP0+rjwtje6xkW7HaXWOFwgRCQeeB55V1c0+mnwMDAeSgGnATODHvj5LVeepaqaqZiYmJgYqckDdelZ/6uqVJz62ZUmNaWvmfriDUO80Ou2RowVCREKARUA1cJuvNqq6U1V3eU9FbQAeAKY7GNNRqd2iuWRUL55fmUORLUtqTJuRd7icf6zJY+bYFJLio9yOExCOFQjxzFo1H0gGpqlqjZ8/qkDbv2f9OL53dn+qaut58hPrRRjTVjz+0Q5E/jeW2B452YN4DBgCXKSqTV62IyJTvWMUiMhgPFc8LXEmojv6JcbyrRG9WLQih0Nl1W7HMcacwP7iCl5elcf0U1Po1bmT23ECxqn7INKAm4BRQH6D+xtmiUiq93Gqt/kU4AsRKQPeBhYDv3Eip5u+f05/KmrqmGdjEcYEvbkf7qBele+d3X57D+C5PyHgVDWH458mim3Q9i7groCHCjIDkuP41oheLFyxmxtP70u3dnhFhDHtwb4jFby0KpfvZqbQp0u023ECyqbaCCJ3TPH0Ip78ZJfbUYwxTZi7dDtK++89gBWIoNI/KY6LvL2Ig3ZFkzFBpyP1HsAKRND5vrcXMc+uaDIm6Dz6oWcdl++d3d/lJM6wAhFk+ifFcfHIXiz8NIfCEutFGBMs8g6X83KWp/fQux1fudSQFYggdMeUAVTV1tlMr8YEkb/+Zzsiwu3ndIzeA1iBCEoZibF8Z0wfFn2WQ36xzdFkjNt2FZXx6po8rhiXSs+EjtF7ACsQQeuOKQOor1f+9uE2t6MY0+H9+f2thIcKt3aAK5casgIRpFK6RnP52BReWpVL7qFyt+MY02FtLShhyfp9XDMxnaS49jnnUlP8KhAiEi0io0Ukzsd7k1o/lgG47Zz+iAh//cB6Eca45U/vbyUmIoybz+hYvQfwo0CIyDggB1gKFIjI3Y2avBOAXAbomdCJK09L49XVeewoLHU7jjEdzsa9xby9IZ/rJ6XTJSbC7TiO86cH8Qfg56qaAEwErhSRxxu8365nWnXbrWf3Iyo8lD/+a6vbUYzpcB55bwudo8O5oZ2u93Ai/hSI4cBTAKq6DpgMDBaRRd71HUwAdY+N5IbTM3hrw3425BW7HceYDmPFjoN8vLWQ753Vn/io9rXWtL/8+YIvB/67XJuqHgXO9772KtaDCLgbT+9Ll+hwHnnP1wJ8xpjWpqo88t5mesRHcdWENLfjuMafAvERcEXDF1S1ErgYCAc6zkXBLomLCufWs/rzybYiPt1R5HYcY9q9f28qYO2eI9zxjQFEhYe6Hcc1/hSIO/CxYI+qVgOXAme3dijzdVdNSKNnQhSPvLsFVXU7jjHtVl298vt/bSGjewzfPbWP23FcdcICoaqFwIhjz0Xk4gbv1arqxyf6DBGJFJH5IpIjIiUislZEph6n/Q9FJF9EikXkaRHp8IsjRIWH8oNvDGBd7hHey853O44x7dbiNXlsLSjlznMHEhbasYdZ/f2vTxKR60XkGjxrSjdXGJALnAkk4FlG9GURSW/cUETOA36KZ2W5dCAD+FUL9tnuTBvTh/5JsTzy7hZq6urdjmNMu1NZU8cf/72VkX0SuPCUnm7HcZ0/90GcAWwDbgBuBLZ6X/Obqpap6v2qultV61X1TWAXcKqP5tcA81U1W1UPAw8C1zZnf+1VWGgIPz1/MDuLyvj7qly34xjT7jyzfDf7iyv56dQhiNj1N/70IPoCaXgGo6O9j/uezE5FJBkYCGT7eHsYsL7B8/VAsoh08/E5c0QkS0SyCgsLTyZSmzFlSBLj0rvy5/e3UVZV63YcY9qNw2XVzF26nXMGJzGh39e+bjokf8YgngWKgee821Hvay0iIuHA88Czqurrus1Y7/6OOfb4a9N8qOo8Vc1U1czExMTGb7dLIsLPLhhMUWkVT9qiQsa0mkc/3E5ZVS0/OX+w21GChr9jEInAn4C/0OCeiOby3li3CKgGbmuiWSkQ3+D5scclLd1vezM6tQsXnNKDeR/v5ECJTQduzMnKPVTOwhU5TBvTh0E9vvZv0Q7L3wIh/O+GuBadmBPPCb35eAa5p6lqTRNNs4GRDZ6PBApU9WBL9tte3X3eYGrq6vm/f9sUHMacrIff3UxICNx57kC3owQVfwtEIZ77IW4HDrRwX48BQ4CLVLXiOO0WArNFZKiIdAHuARa0cJ/tVnr3GK4an85Lq3LZnH/U7TjGtFmrcw7z5hf7mXN6RodaDMgf/lzFdDWe0zxXerd472t+E5E04CZgFJAvIqXebZaIpHofpwKo6rvAI8CHeGaRzQHua87+OorvT+lPXFQ4v37rS7t5zpgWUFUeemsTiXGR3HRmx5vO+0TC/GiT4/3fckAbPPebquZw/FNTsY3a/xH4Y3P309F0jo7g+1MG8OCbm1i6tZCzByW5HcmYNuWtDftZu+cIj0wbQUykP1+HHYs/VzF9hOeS1Ke820DvayYIXDU+jfRu0fzmrS+ptZvnjPFbZU0dD7+7mcE94pjWwafUaEpzxiB2quoC7+P/EpGZrR3K+C8iLISfTh3CtgOlvPj5HrfjGNNmPLN8N7mHKrjnwqGEhthNcb74VSBU9XXgVRF5GHgLQEQ6i8hL2DQYrjtvWDITMrrxx39v5Uh5tdtxjAl6B0oq+dsH2/jm0GQmD+judpyg1ZyZqEbiGWReJSKzgQ3AEWB0IIIZ/4kIv7xoKMUVNfzpfVu/2pgT+d27W6iuq+cXFwxxO0pQ87tAqOo+4Nven5kHvKOqN6lqWaDCGf8N6RnPzHGpLPosh20Fdk+hMU1Zn3uEV1bncf3kvqR3j3E7TlDzu0CIyCggC9gJXAKcIyIvikjnQIUzzXPnNwcSExHKA29usstejfFBVfnVG9l0j43ktrP7ux0n6DXnFNN/gD+q6re9s7GOxHPp64aAJDPN1i02kh98YyCfbCvi/S9bej+jMe3XknX7WLPnCHefP4i4DrrOdHM0p0CMVdX5x554p/CeDXyv9WOZlrpqQhoDkmJ54M1sKmvq3I5jTNAoqazh129/ycg+CUwfY5e1+qM5YxA+pw5V1X+2XhxzssJDQ/jVxcPIPVTBEx/ZbK/GHPOX/2yjqLSKBy4ZTohd1uqXjr2eXjs1sX93LhzRk7lLt5N7qNztOMa4bltBCc8s383lmSmMTLFhU39ZgWin7rlwCCEiPPTWJrejGOMqVeX+N7KJjgjlx+cNcjtOm2IFop3qmdCJ26f0573sApZusQFr03G9vSGf5dsP8uPzBtEtNtLtOG2KFYh27IbJGWQkxnDfP23A2nRMJZU1PPBmNsN6xXPFaWlux2lzrEC0YxFhITx0yXByDpYz98PtbscxxnF//PdWDpRU8etLT7H5llrACkQ7N7F/d749qhePf7STHYWlbscxxjEb9xbz7Ke7mXVaKqNsYLpFHCsQInKbiGSJSJWILDhOu2tFpK7BokKlInKWUznbo19cOJTI8BDufX2j3WFtOoT6euWe1zfSNSaCH5832O04bZaTPYh9wEPA0360XaGqsQ22pYGN1r4lxkVy9/mD+XTHQZas2+d2HGMC7oXP97Au9wj3XDiUhE52x3RLOVYgVHWxd9rwg07t0/zPFeM83ewH39zE4TKbEty0XwVHK3n4nc1M6t+NS0b1cjtOmxasYxCjRaRIRLaKyL0i4nMtQBGZ4z1tlVVYWOirifEKDRF+O+0Uiis80w0Y017dtySb6rp6fv3tUxCxgemTEYwF4mNgOJAETANmAj/21VBV56lqpqpmJiYmOhixbRrcI56bzszg1dV5LN9e5HYcY1rde9n5vJudzx3fGGBTebeCoCsQqrpTVXepar2qbgAeAKa7nau9uP2cAaR3i+bnr22weyNMu1JSWcN9S7IZ3COOG0/PcDtOuxB0BcIHBayf2EqiwkP5zXdOIedgOf/3/la34xjTah5+dzMFJZX8dtoIwkPbwldb8HPyMtcwEYkCQoFQEYnyNbYgIlNFJNn7eDBwL7DEqZwdwcR+3ZkxNoUnP97J+twjbscx5qSt2HGQ5z7bw3UT+9o9D63IyTJ7D1AB/BS40vv4HhFJ9d7rkOptNwX4QkTKgLeBxcBvHMzZIfz8wiEkxUVx96tfUF1b73YcY1qsvLqWn/zjC9K6RdtkfK3Myctc71dVabTdr6p7vPc67PG2u0tVk1U1RlUzVPWXqlrjVM6OIj4qnF9fOpwtBSX8zabhMG3YH/61lT2Hyvntd0bQKSLU7Tjtip2o68CmDEnm0tG9mfvhdjbtO+p2HGOabXXOYZ5evosrx6cyoV83t+O0O1YgOrhffmsonaPDueuV9XaqybQpFdV1/PiV9fRK6MRPpw5xO067ZAWig+sSE8GvLz2FTfuP8tcPtrkdxxi/PfLeZnYWlfHI9BHERvq8l9acJCsQhvOG9eA7Y3ozd+kO1tlVTaYN+HRHEc8s3801E9KY1L+723HaLSsQBoD7LhpGUlwkP3p5nd1AZ4JaSWUNP37lC9K7RfOTqTZTayBZgTAAJHQK5+FpI9hRWMbv3tvidhxjmvTQm1+yv7iCP1w2kugIO7UUSFYgzH+dMTCRq8anMX/ZLpZts7maTPB5d2M+L2XlctOZ/Tg1ravbcdo9KxDmK35+wRD6Jcbwo1fWcaTcpgU3wePA0Up+tvgLhveO54ffGOh2nA7BCoT5ik4Rofx5xmgOlVXz89c22Ap0JijU1yt3vfoFFTV1/Ony0USE2VeXE+wom68Z3juBO785iLc35PPq6jy34xjDsyt28/HWQn5x4VD6J8W6HafDsAJhfJpzRgan9e3Kff/MZmdhqdtxTAe2ad9R/t87mzlncBJXnpZ64h8wrcYKhPEpNET404xRRISFcPuLa6mqtUtfjfPKq2u57cU1dO4Uzu+mj7AV4hxmBcI0qWdCJ343fSTZ+47y23c2ux3HdED3LclmV1EZf5oxim6xkW7H6XCsQJjj+ubQZK6dmM4zy3fz/qYCt+OYDmTJur28sjqP287uz8R+dre0G5xcMOg2EckSkSoRWXCCtj8UkXwRKRaRp0XE/ungop9dMJihPeO569X15B0udzuO6QB2FJby88UbyEzrwh1TBrgdp8NysgexD3gIePp4jUTkPDyLCk0B0oEM4FeBDmeaFhkWyqOzxlBbp9z2wlqb9dUEVEV1Hbc+t4aIsBD+MnM0YbZ8qGucXDBosaq+Dhw8QdNrgPmqmq2qh4EHgWsDnc8cX9/uMfxu+gjW5R7hN29/6XYc047du2QjWw+U8KcZo+nVuZPbcTq0YCzNw4D1DZ6vB5JFxFYDcdnUU3py3aR0Fny6m7e+2O92HNMOvZyVy6ur87j97P6cOTDR7TgdXjAWiFiguMHzY4/jGjcUkTnecY2swsJCR8J1dD+bOoTRqZ25+9X1bD9Q4nYc045s3FvMva9vZGK/btxhU2kEhWAsEKVAfIPnxx5/7dtIVeepaqaqZiYm2r82nBARFsLcWWPoFBHKnEWrKam05cLNyTtUVs1Ni1bTNSaCv8wcTWiI3e8QDIKxQGQDIxs8HwkUqOqJxi6MQ3omdOJvV4wh52A5d768nvp6m6/JtFxtXT23v7iGwtIqHr/yVLrb/Q5Bw8nLXMNEJAoIBUJFJEpEfE3mvhCYLSJDRaQLcA+wwKmcxj/jM7rxiwuG8O9NBfztw+1uxzFt2O/e28Ly7Qd56NvDGZnS2e04pgEnexD3ABV4LmG90vv4HhFJFZFSEUkFUNV3gUeAD4Ec73afgzmNn66blM6lo3vzf8PDN34AABHlSURBVO9v5V/Z+W7HMW3Q62v38sTHO7lyfCqXZaa4Hcc0Iu1lOufMzEzNyspyO0aHU1lTx+VPrGDbgVL+cctEhvSMP/EPGQOs3XOYy+d9xpjUziyafRrhdr+DK0Rktapm+nrP/h8xJyUqPJR5V2cSFxXGDc9mUVRa5XYk0wbsL65gzqLV9IiP4rFZp1pxCFL2/4o5acnxUTx5dSYHy6q4edFqm/nVHFd5dS03LsyiorqOp67JpEtMhNuRTBOsQJhWMaJPZ37/3ZFk5Rzm7le/sJXojE919cr3X1zHpn1H+cvMUQxM/trtTSaI+LqKyJgW+daIXuw5VM4j724htWs0Pzp3kNuRTJB58M1NvP9lAQ9cMoxzBie7HcecgBUI06puObMfew6W89cPtpPSJZrLxtqVKcbj6WW7WPDpbmZP7svVE9LdjmP8YAXCtCoR4cFvD2fvkQp+/toGkuIjOWtQktuxjMve2bCfB9/axHnDkvn5BUPcjmP8ZGMQptWFh3qm4xjUI45bnlvD2j2H3Y5kXPTpjiLu+Ps6xqR24U+X2zQabYkVCBMQcVHhLLhuHEnxkVy/YBXbD5S6Hcm4YOPeYuYsXE1at2jmX5NJp4hQtyOZZrACYQImMS6ShdePIzREuObpz9l3pMLtSMZBOQfLuPaZVcRHhbFw9jg6R9vlrG2NFQgTUGndYlhw3TiOVtRw5VMrKSyxG+k6gn1HKrjiyZXU1dfz7PXj6JlgC/+0RVYgTMAN753AM9eNZX9xJVfNX8mR8mq3I5kAOlBSyaynVnK0ooZFs09jgN3r0GZZgTCOyEzvypNXZ7KzsIxrnv7c1pFopw6XVXP1/M/JL65kwfVjGd47we1I5iRYgTCOmTygO3NnjSF731GutiLR7hwuq2bWUyvZWVTGU9dkcmpaV7cjmZNkBcI46htDk/nbFWPYkFfM1U9/zlErEu3CobJqrnhqJdsLS3nq6kwm9e/udiTTCqxAGMedP7wHj87yFon5ViTaukPHeg7e4nDGQFv+t71wckW5riLymoiUiUiOiFzRRLv7RaTGu4jQsS3DqZzGGecN6+E93VTMFU9+xkGbJrxNKjhayeVPrGBnYSlPWnFod5zsQTwKVAPJwCzgMREZ1kTbl1Q1tsG207GUxjHnDuvBvKsz2VZQyuXzPqPgaKXbkUwz5B4q57uPr2DfkQoWXDfOikM75EiBEJEYYBpwr6qWquoy4J/AVU7s3wSvswcl8ez148gvrmT645+y52C525GMH7YfKOG7j6+guKKG528cz4R+3dyOZALAqR7EQKBOVbc2eG090FQP4iIROSQi2SJyS1MfKiJzRCRLRLIKCwtbM69x0PiMbjx/w2mUVNbynceWsyGv2O1I5jhW7T7EtMdWUFuvvHTTeEaldHY7kgkQpwpELND4r74Y8HUHzcvAECARuBH4pYjM9PWhqjpPVTNVNTMx0bq3bdnIlM68evNEIsNCuXzeCj7aagU/GL27MZ8rn1pJt5gIXrt1IoN72Brk7ZlTBaIUaPybFA+UNG6oqptUdZ+q1qnqp8CfgekOZDQu658Uy+JbJ5LWLYbZC1bx8qpctyMZL1VlwfJd3PL8aob2iufVWyaS0jXa7VgmwJwqEFuBMBEZ0OC1kUC2Hz+rgM0P3EEkx0fx8k2ec9p3/+MLfvP2l9TV2/Klbqqpq+feJRu5/41NTBmczAs3jKerrSPdIThSIFS1DFgMPCAiMSIyCbgEWNS4rYhcIiJdxGMc8H1giRM5TXCIiwrn6WvHctX4NOZ9vJObFq2mtKrW7VgdUnFFDdcvWMVzn+3hpjMyeOKqU23K7g7EyctcbwU6AQeAF4FbVDVbRE4XkYaLBcwAtuM5/bQQeFhVn3UwpwkC4aEhPPjt4fzq4mF8sLmA78xdzs5CW1PCSVvyS7jkb8v4bOdBHpk+gp9dMMQW++lgRLV9dN8zMzM1KyvL7RgmAJZtK+L2F9dQW6f88fJRfHOoLXYfaG+s38fdr35BbFQYc2eNYWy6zavUXonIalXN9PWeTbVhgt7kAd154/bJpHWP5saFWfzuvc3U1tW7Hatdqqqt41dvZHP7i2sZ1iuet26fbMWhA7MCYdqEPl2iefXmiVyemcKjH+5gxrzP2Gsr1LWq3UVlTH9sBc8s3821E9N54cbxJMVHuR3LuMgKhGkzosJDeXj6CP48YxSb80u44M+f8O7G/W7HavNUldfX7uVbf13GnkPlPHHVqdx/8TAiwuzroaOz3wDT5lwyqjdvfX8yad2iufm5NfzwpXUUV9iMsC1xsLSKW59fww9eWsfgHnG8fcfpnDesh9uxTJAIczuAMS2R1i2Gf9wykb99sJ2/fbidFTsO8vD0EZxpE8b57b3sfH7x2gaOVtTyk/MHM+eMDLtKyXyF9SBMmxUeGsIPvzmQ126dSGxUGNc8/Tl3/H0tRTZ1+HHlF1dy86LV3LRoNYlxUfzz9kncclY/Kw7ma+wyV9MuVNbUMXfpDh5bup3oiDB+NnUwl2WmEGJfev9VW1fP8yv38Lv3tlBTV88d3xjAjadnEB5q/07syI53masVCNOubCso4eevbWDV7sOc0juB+y4aSqZdpsny7UU88MYmthSUMLl/d3596XDSusW4HcsEASsQpkNRVZas28dv39lM/tFKvjWiJ3edO4j07h3vC3FbQQm/e28L/9pUQJ8unfjFBUM4f3gPRKxnZTysQJgOqby6lseX7uDJT3ZRXVfPZZkp3DFlAD0S2v+1/bmHyvnT+9t4bW0e0RFh3HxmBjecnkFUuM2jZL7KCoTp0A6UVPLoB9t54fM9iAjTT+3DTWdktMtTLNsPlPL4Rzt4fe1eQkKEayakcctZ/W32VdMkKxDG4PlX9dylO/jH6jxq6+u54JSeXDepL2NSO7fpUy6qyspdh1iwfDfvbconMiyEGWNTuenMDHomdHI7nglyViCMaeDA0UrmL9vFCyv3UFJVy7Be8Vw9IY0LR/QiNrLt3Bp0tLKGf67bx6IVOWwpKCGhUzhXjU/juknpdIuNdDueaSOsQBjjQ1lVLa+t3cvCFbvZWlBKp/BQzh/eg0tH92ZCv25BeflndW09y7YXsnjNXv61qYDq2nqG9ozn2onpXDSyl63VYJrNCoQxx6GqrM45zOK1e3lz/T6OVtaS0CmcKUOSOHdoDyb170ZcVLhr+YrLa1i2vYj3svP5cPMBSqpq6RIdzsUje3HpmD6M7JPQpk+RGXcFRYEQka7AfOBcoAj4maq+4KOdAL8FbvC+NB/4iZ4gqBUI0xoqa+pYuqWQf23K5z9fHqC4oobQEGFknwQm9e/OmLQujOzTOaCDvkWlVazPPcLqnMMs317Ehr3F1Ct0jYngG96idcbARJtMz7SK4xUIJ0+4PgpUA8nAKOAtEVmvqo3XpZ4DfBvPmtUK/BvYCTzuYFbTQUV5TzOdP7wHNXX1ZO32fEkv31HE3KU7/rs+dkrXTgxKjqd/Uiz9EmPo0yWaHglR9IiP8us0T1lVLflHKykoriTvcAU7CkvZUVjKl/tL/juNeViIMDq1M7efM4DJA7ozOqUzYUF42su0X470IEQkBjgMDFfVrd7XFgF7VfWnjdp+CixQ1Xne57OBG1V1/PH2YT0IE2ilVbVs3FvM+twjrM87wraCUnYfLKOm7qt/Q5FhIcRFhRETGUaE9wtd8YwflFXVUlJVS3XtVxc8iggLIaN7DP2TYhnZpzOjUjszrFc80RFtZ9DctE3B0IMYCNQdKw5e64EzfbQd5n2vYbthvj5URObg6XGQmpraOkmNaUJsZBjjM7oxPqPbf1+rrasn93AF+45UkF9cSf7RSo5W1FBSVUtpZS219f8rBOGhIcRGhhEbFUbnThH0SIgkOT6K3p070adLtE2WZ4KOUwUiFihu9FoxEOdH22IgVkSk8TiEt5cxDzw9iNaLa4x/wkJD6Ns9hr4dcBoP0/45dUKzFIhv9Fo8UOJH23ig9ESD1MYYY1qXUwViKxAmIgMavDYSaDxAjfe1kX60M8YYE0COFAhVLQMWAw+ISIyITAIuARb5aL4QuFNEeotIL+BHwAInchpjjPkfJ6+ZuxXoBBwAXgRuUdVsETldREobtHsCeAPYAGwE3vK+ZowxxkGOXUOnqofw3N/Q+PVP8AxMH3uuwN3ezRhjjEvsrhtjjDE+WYEwxhjjkxUIY4wxPrWb2VxFpBDIaeGPd8czgWCwsVzNY7maL1izWa7mOZlcaaqa6OuNdlMgToaIZDU1F4mbLFfzWK7mC9Zslqt5ApXLTjEZY4zxyQqEMcYYn6xAeMxzO0ATLFfzWK7mC9Zslqt5ApLLxiCMMcb4ZD0IY4wxPlmBMMYY45MVCGOMMT51uAIhIpEiMl9EckSkRETWisjUE/zMD0UkX0SKReRpEYkMULbbRCRLRKpEZMEJ2l4rInUiUtpgO8vtXN72Th2vriLymoiUef//vOI4be8XkZpGxyvD6Szi8bCIHPRuj4hIwNYabUaugB6fRvtqzu+5I79Lzc3m8N9fs76zWvOYdbgCgWcG21w862EnAPcCL4tIuq/GInIe8FNgCpAOZAC/ClC2fcBDwNN+tl+hqrENtqVu53L4eD0KVAPJwCzgMRHxuX6510uNjtdOF7LMwTOr8UhgBPAt4KZWzNHSXBDY49OQX79PDv8uNSubl1N/f35/Z7X6MVPVDr8BXwDTmnjvBeA3DZ5PAfIDnOchYMEJ2lwLLHP4OPmTy5HjBcTg+eIb2OC1RcBvm2h/P/BcgI6L31mAT4E5DZ7PBj4LglwBOz4t/X1y42+vGdkc//trtH+f31mtfcw6Yg/iK0QkGRhI08uaDgPWN3i+HkgWkW6BzuaH0SJSJCJbReReEXFsfY/jcOp4DQTqVHVro30drwdxkYgcEpFsEbnFpSy+js/xMjuVCwJ3fFoqmP/2wKW/vxN8Z7XqMevQBUJEwoHngWdVdXMTzWKB4gbPjz2OC2Q2P3wMDAeSgGnATODHribycOp4Nd7PsX01tZ+XgSFAInAj8EsRmelCFl/HJzZA4xDNyRXI49NSwfq3By79/fnxndWqx6zdFQgRWSoi2sS2rEG7EDzd7WrgtuN8ZCkQ3+D5scclgcjlL1Xdqaq7VLVeVTcADwDTm/s5rZ0L545X4/0c25fP/ajqJlXdp6p1qvop8GdacLya0Jwsvo5PqXrPB7Qyv3MF+Pi0VKv8LgVCa/39NYef31mteszaXYFQ1bNUVZrYJoPnShJgPp6Bu2mqWnOcj8zGM6B4zEigQFUPtnauk6RAs/8VGoBcTh2vrUCYiAxotK+mThV+bRe04Hg1oTlZfB0ffzMHMldjrXl8WqpVfpccEtDj1YzvrFY9Zu2uQPjpMTzd6YtUteIEbRcCs0VkqIh0Ae4BFgQilIiEiUgUEAqEikhUU+c1RWSq91wkIjIYz5UNS9zOhUPHS1XLgMXAAyISIyKTgEvw/AvL13/DJSLSRTzGAd+nlY5XM7MsBO4Ukd4i0gv4EQH6fWpOrkAeHx/78vf3ybG/veZmc/Lvz8vf76zWPWZujcK7tQFpeKp9JZ7u2LFtlvf9VO/z1AY/cydQABwFngEiA5Ttfm+2htv9vnIBv/dmKgN24unihrudy+Hj1RV43XsM9gBXNHjvdDynbo49fxE46M26Gfi+E1l85BDgEeCQd3sE75xoTh4jp4+PP79Pbv4uNTebw39/TX5nBfqY2WR9xhhjfOqop5iMMcacgBUIY4wxPlmBMMYY45MVCGOMMT5ZgTDGGOOTFQhjjDE+WYEwxhjjkxUIY4wxPlmBMMYY45MVCGMCQET6eddWGON93su7dsBZLkczxm821YYxASIiN+KZF+dU4DVgg6re5W4qY/xnBcKYABKRfwJ98Uy2NlZVq1yOZIzf7BSTMYH1JJ6Vx/5qxcG0NdaDMCZARCQWz5rAHwJTgVNU9ZC7qYzxnxUIYwJEROYDcap6mYjMAzqr6mVu5zLGX3aKyZgAEJFLgPOBm70v3QmMEZFZ7qUypnmsB2GMMcYn60EYY4zxyQqEMcYYn6xAGGOM8ckKhDHGGJ+sQBhjjPHJCoQxxhifrEAYY4zxyQqEMcYYn/4/4/tCvGNWZU4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_function(f, 'x', 'x**2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEMCAYAAADeYiHoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3yV5f3/8dcnm0xWEmYSwh6yDMhy0qpo1VqogrhRHNXaWrVDrdb1q3Z8OxQVRREcdRSlbmsVFUQkbIJsCGEkJIyQPT+/P86hjfEETkLOfZ8kn+fjcT96xpVzv72bnA/Xfd33dYmqYowxxtQX4nYAY4wxwckKhDHGGJ+sQBhjjPHJCoQxxhifrEAYY4zxKcztAM2lc+fOmpaW5nYMY4xpUVasWFGgqom+3ms1BSItLY3MzEy3YxhjTIsiItkNvWenmIwxxvhkBcIYY4xPViCMMcb4ZAXCGGOMT1YgjDHG+OR4gRCRviJSLiIvNvC+iMijInLAuz0mIuJ0TmOMaevcuMz1CWD5Md6fCfwQGAYo8G9gO/BU4KMZY4w5ytEehIhMBQ4D/zlGs6uAP6nqblXdA/wJuDpQmVbnHObRDzYG6uONMSZgVJWH391A1t7CgHy+YwVCROKBB4BfHKfpYGBNnedrvK/5+syZIpIpIpn5+flNyrVu92GeXLSN9XsCc4CNMSZQvtp+kGe+2MGm3KKAfL6TPYgHgTmqmnOcdrFA3W/rQiDW1ziEqs5W1QxVzUhM9Hmn+HFdOLw7kWEh/GP5rib9vDHGuOXV5buIiwpj0pCuAfl8RwqEiAwHvgf8nx/Ni4H4Os/jgWIN0NJ3Ce3COe+krixcvZeyyppA7MIYY5pdYWkV76/P5aLh3WgXERqQfTjVgzgDSAN2iUgucAcwWURW+mibhWeA+qhh3tcC5pKMnhSVV/P++n2B3I0xxjSbhWv2UFFdy9RRKQHbh1MFYjbQGxju3Z4C3gXO8dF2HnC7iHQXkW54xizmBjLcmPSOpHWK5tXlxzv7ZYwxweHV5TkM6hrPkO4JAduHIwVCVUtVNffohuc0Urmq5ovIqSJSXKf508DbwDpgPZ5C8nQg84kIP87oybIdB9meX3z8HzDGGBet31NI1t4jTB3dM6D7ceVOalW9X1Uv9z7+QlVj67ynqnqXqnb0bncFavyhrikn9yA0RHgtc3egd2WMMSfkH8t3ERkWwkXDugd0PzbVhldyfBRn9k/kjRW7qaqpdTuOMcb4VFZZw8LVe5k0pAsJ0eEB3ZcViDouHZVCQXEFn2zc73YUY4zx6b11+ygqr+aSUYE9vQRWIL7lzP6JJMdH8srXdk+EMSY4vfL1Lnp1jmFseqeA78sKRB1hoSFcktGTzzbns+dwmdtxjDHmW7bkFZGZfYipo3rixBymViDquSTD022zS16NMcHmla9zCA8VJp/cw5H9WYGop2fHaE7rm8jrmTlU22C1MSZIlFfVsGDVbs4e3IXOsZGO7NMKhA/TRvdkX2E5n21u2gSAxhjT3D7MyuVwaRXTAnjndH1WIHyYODCZzrGRvPK1nWYyxgSHV77eRUrHaMb1Dvzg9FFWIHwIDw3hkowefLIxj9zCcrfjGGPauO35xXy1/SBTR/ckJMS5BTatQDRg6qgUatUGq40x7nvl612EhQhTHBqcPsoKRANSOkVzWr9E/rF8lw1WG2NcU15Vw+srdnPO4C4kxUU5um8rEMcw/ZQU9hWW8+kmG6w2xrjj/fX7OFxaxWWnODc4fZQViGOYOCCJ5PhIXlqW7XYUY0wb9dJXzt05XZ8ViGMICw3h0lEpfLY5n5yDpW7HMca0MRtzj5CZfYjLRqc4Ojh9lBWI45g6qicCNj+TMcZxLy/bRURoiGN3TtfnWIEQkRdFZJ+IHBGRzSJyXQPtrhaRGhEprrOd4VTO+rq1b8dZA5J5LXM3ldU2WG2McUZpZTVvrtzDeSd1oWNMhCsZnOxB/D8gTVXjgQuBh0Tk5AbaLlXV2DrbIsdS+jB9jGca8A+zct2MYYxpQ/61ei9FFdVcdkqqaxkcKxCqmqWqFUeferfeTu3/RJzWN5GeHdvx4lc2WG2MCTxVZd7SbPonxzEqrYNrORwdgxCRWSJSCmwE9gHvNdB0hIgUeE9F3SsiYQ183kwRyRSRzPz8wF2KGhoiTD8llWU7DrI5ryhg+zHGGIBVOYfZsO8Il49NdWRa74Y4WiBU9WYgDjgVWABU+Gj2OTAESAImA9OAOxv4vNmqmqGqGYmJiYEJ7XVJRk8iwkKsF2GMCbgXl2YTGxnGxSMCu+b08Th+FZOq1qjqYqAHcJOP97er6g5VrVXVdcADwBSnc9bXMSaCH5zUlQUr91BcUe12HGNMK3WwpJJ31u7jRyO7Exvp8+SJY9y8zDUM/8YgFHCvj1XH5WNTKa6o5q1Ve9yOYoxppV7LzKGyppbLx7g3OH2UIwVCRJJEZKqIxIpIqIicg+fU0Sc+2k4SkWTv4wHAvcBCJ3Iez4ie7RncLZ4Xv8pGVd2OY4xpZWpqlZeWZXNKr470S45zO45jPQjFczppN3AI+CPwM1VdKCIp3nsdjk40MhFYKyIleAaxFwCPOJTzmESEK8aksjHXsy6sMcY0p88355NzsIwrxrrfewDPaZ6AU9V84PQG3tsFxNZ5fgdwhxO5muLC4d145L1veOHLnYxK6+h2HGNMK/LC0p0kxkVy9qAubkcBbKqNRouOCOOSjJ58sD6XvCO2mJAxpnnsLChh0aZ8pp+SQkRYcHw1B0eKFuaKsanUqPLSMpufyRjTPOYtzSY8VFyZ1rshViCaILVTDGf2T+LlZbtsfiZjzAkrqajm9RU5TBrS1fFFgY7FCkQTXTk2lYLiCt5fv8/tKMaYFu7NVXsoKq/mqnHBMTh9lBWIJjqtbyK9Osfwwpc73Y5ijGnBPPMu7WRI93hGprg375IvViCaKCTEc8nryl2HWbe70O04xpgWaun2A2zOK+bKsWmuzrvkixWIEzAlowfREaE8/+UOt6MYY1qouUt20iE6nAuHdXM7yndYgTgB8VHhTDm5B++s2Ud+ka95B40xpmE5B0v5+Js8po1OISo81O0432EF4gRdNS6NyppaXrZLXo0xjTRv6U7PDA1Bcud0fVYgTlDvxFjO6J/Ii8uy7ZJXY4zfSiqq+cfyHCYN6ULXhHZux/HJCkQzuHpcGvlFFby7bq/bUYwxLcSClbspKq/mmvFpbkdpkBWIZnBa30TSE2N4fslOm+XVGHNctbXK81/uZGiPhKC7tLUuKxDNICREuGZcGmt3F7Jyl83yaow5ti+2FrA9v4Rrxgffpa11WYFoJj8a2YP4qDCeW7LT7SjGmCD33OIdJMZFcv5JwXdpa11WIJpJTGQY00an8P66feQcLHU7jjEmSG3JK+KzzflcOSY1aGZtbYhj6UTkRRHZJyJHRGSziFx3jLY/F5FcESkUkedEJNKpnCfiqnGe7qJNv2GMachzS3YQGRbC9CBYUvR4nCxf/w9IU9V44ELgIRE5uX4j73Kkv8KzslwakA78zsGcTdatfTvOO6krry7Poai8yu04xpggc6C4gn+u3MOPRvagY0yE23GOy7ECoapZqnr0dmP1br19NL0KmONtfwh4ELjamZQnbsaEXhRVVPNa5m63oxhjgsxL3iUCZkxIczuKXxw9ASYis0SkFNgI7MOz5nR9g4E1dZ6vAZJFpJOPz5spIpkikpmfnx+QzI01vGd7MlI78PySHdTU2iWvxhiPiuoa5i3N5oz+ifRJinM7jl8cLRCqejMQB5wKLAB8TWAUC9SdHvXo4+8cUVWdraoZqpqRmJjY3HGbbMaEXuw+VMZHWbluRzHGBIl/rd5LQXEFMyb0cjuK3xwfQlfVGlVdDPQAbvLRpBiIr/P86OOiQGdrLmcP7kLPju14drHN8mqM8az5MGfxDvonxzGhT2e34/jNzWuswvA9BpEFDKvzfBiQp6oHHEnVDEJDhBnje7Ei+xArsg+6HccY47LPtxSwMbeI609LD+ob4+pzpECISJKITBWRWBEJ9V6pNA34xEfzecAMERkkIh2Ae4C5TuRsTj/O6ElCu3Bmf77d7SjGGJc98/l2kuMjg3LNh2NxqgeheE4n7QYOAX8EfqaqC0UkRUSKRSQFQFU/AB4DPgWyvdt9DuVsNjGRYVwxJpWPNuSxo6DE7TjGGJdk7S1k8dYCrhnfK+hvjKvPkbSqmq+qp6tqe1WNV9WTVPUZ73u7VDVWVXfVaf9nVU32tr2mzuWxLcqV41IJDwlhzmLrRRjTVj3z+XZiIkKZNjrF7SiN1rLKWQuTFBfFxSO683rmbg4Ut8gaZ4w5AXsPl/H22n1MHZ1CQrtwt+M0mhWIALvu1F5UVNcy/6tst6MYYxz2/BLPlYzBvObDsViBCLC+yXFMHJDEvKXZlFXWuB3HGOOQwtIqXl62ix8M7UqPDtFux2kSKxAOuPGM3hwsqeS1zBy3oxhjHPLismxKKmu44TRfV/O3DFYgHDAqrSMnp3bgmS+2U11j61Yb09qVV9Xw/JIdnNYvkUHd4o//A0HKCoRDbjy9N7sPlfHuun1uRzHGBNgbK3ZTUFzJjaenux3lhFiBcMjEAUn0TYrlqc+227rVxrRiNbXKM19sZ1iPBMamf2eO0RbFCoRDQkKEmael882+I3y2OThmnjXGNL/31+8j+0ApN57eu0VNq+GLFQgHXTS8O10Tonhy0Ta3oxhjAkBVeeqzbfTqHMPZg7u4HeeEWYFwUERYCNedms6yHQdZkX3I7TjGmGb2+ZYC1u85wo2npxMa0rJ7D2AFwnHTRvekQ3Q4Ty7a6nYUY0wzm/XpVromRHHxiB5uR2kWViAcFh0RxrXje/HxN/v5Zt8Rt+MYY5pJ5s6DLNtxkOtPTW9xk/I1pHX8V7QwV45NIzYyzMYijGlFZi3aRseYCKaO7ul2lGZjBcIFCdHhTB+Twjtr97LTpgI3psXL2lvIJxv3c824NKIjwtyO02ycWjAoUkTmiEi2iBSJyCoRmdRA26tFpMa7RsTR7QwncjppxoRehIWG8NRn1oswpqV7ctE2YiPDuHJsmttRmpVTPYgwIAc4HUgA7gVeE5G0Btov9a4RcXRb5EhKByXFRTF1VE/+uXI3ew6XuR3HGNNEW/cX8+66fVw+JpWE6JY3pfexOLVgUImq3q+qO1W1VlXfAXYAJzux/2B1w+meSbyetl6EMS3WrE+3EhkWwnWn9nI7SrNzZQxCRJKBfkBWA01GiEiBiGwWkXtFxOdJPRGZKSKZIpKZn9/y7k7u3r4dk0f24B/Lc9h/pNztOMaYRso+UMLCNXu5/JRUOsdGuh2n2TleIEQkHHgJeEFVN/po8jkwBEgCJgPTgDt9fZaqzlbVDFXNSExMDFTkgLr5jD7U1CpPf27LkhrT0sz6dBuh3ml0WiNHC4SIhADzgUrgFl9tVHW7qu7wnopaBzwATHEwpqNSOkVz0fBuvLQsmwJbltSYFmP3oVL+uXI300b1JCk+yu04AeFYgRDPrFVzgGRgsqpW+fmjCrT8e9aP4Sdn9qGiupZnvrBehDEtxVOfbUPkf2OJrZGTPYgngYHABara4GU7IjLJO0aBiAzAc8XTQmciuqN3Yiw/GNqN+UuzOVhS6XYcY8xx7Css47Xlu5lyck+6tW/ndpyAceo+iFTgBmA4kFvn/obpIpLifZzibT4RWCsiJcB7wALgESdyuumnZ/WhrKqG2TYWYUzQm/XpNmpV+cmZrbf3AJ77EwJOVbM59mmi2Dpt7wDuCHioINM3OY4fDO3GvKU7uf7UXnRqhVdEGNMa7D1cxqvLc/hxRk96dIh2O05A2VQbQeS2iZ5exDNf7HA7ijGmAbMWbUVp/b0HsAIRVPokxXGBtxdxwK5oMibotKXeA1iBCDo/9fYiZtsVTcYEnSc+9azj8pMz+7icxBlWIIJMn6Q4LoyvZN5/viE/tiOkpcFLL7kdy5g2b/ehUl7L9PQeurfiK5fqsgIRbF56idsev5OKkDCeOmUyZGfDzJlWJIxx2d//sxUR4daz2kbvAaxABJ+77yZ97zZ+lPUJ80ecR25sJygthbvvdjuZMW3WjoIS3li5m8tGp9A1oW30HsAKRPDZtQuA25b8g1oJ4fGxl3zrdWOM8/768WbCQ4Wb28CVS3VZgQg2KZ77BXsW5nHp2o94ddjZ5MQn/fd1Y4yzNucVsXDNXq4al0ZSXOucc6khfhUIEYkWkREiEufjvfHNH6sNe/hhiPZcPnfL0lcRVf5+2uWe140xjvvLx5uJiQjjxtPaVu8B/CgQIjIayAYWAXkicle9Ju8HIFfbNX06zJ4Nqal0LT7I5dsW88bgM9l29kVuJzOmzVm/p5D31uVy7fg0OsREuB3Hcf70IP4E/EZVE4BxwOUi8lSd91v1TKuumD4ddu6E2lpunv8IURFh/PmjzW6nMqbNeezDTbSPDue6Vrrew/H4UyCGAM8CqOpqYAIwQETme9d3MAHUOTaS605N5911+1i3u9DtOMa0GUu3HeDzzfn85Iw+xEe1rrWm/eXPF3wp8N/l2lT1CHCu97U3sB5EwF1/ai86RIfz2Ie+FuAzxjQ3VeWxDzfSJT6KK8amuh3HNf4UiM+Ay+q+oKrlwIVAONB2Lgp2SVxUODef0YcvthTw5bYCt+MY0+r9e0Meq3Yd5rbv9SUqPNTtOK7xp0Dcho8Fe1S1ErgYOLO5Q5nvumJsKl0Tonjsg02oqttxjGm1amqVP360ifTOMfz45B5ux3HVcQuEquYDQ48+F5EL67xXraqfH+8zRCRSROaISLaIFInIKhGZdIz2PxeRXBEpFJHnRKTNL44QFR7Kz77Xl9U5h/kwK9ftOMa0WgtW7mZzXjG3n92PsNC2Pczq7399kohcKyJX4VlTurHCgBzgdCABzzKir4lIWv2GInIO8Cs8K8ulAenA75qwz1Zn8sge9EmK5bEPNlFVU+t2HGNanfKqGv78780M65HA+Sd1dTuO6/y5D+I0YAtwHXA9sNn7mt9UtURV71fVnapaq6rvADuAk300vwqYo6pZqnoIeBC4ujH7a63CQkP41bkD2F5Qwj+W57gdx5hW5/klO9lXWM6vJg1ExK6/8acH0QtIxTMYHe193OtEdioiyUA/IMvH24OBNXWerwGSRaSTj8+ZKSKZIpKZn59/IpFajIkDkxid1pG/fryFkopqt+MY02ocKqlk1qKtnDUgibG9v/N10yb5MwbxAlAIvOjdjnhfaxIRCQdeAl5QVV/XbcZ693fU0cffmeZDVWeraoaqZiQmJtZ/u1USEX593gAKiit4xhYVMqbZPPHpVkoqqvnluQPcjhI0/B2DSAT+AvyNOvdENJb3xrr5QCVwSwPNioH4Os+PPi5q6n5bmxEpHTjvpC7M/nw7+4vK3Y5jTIuXc7CUeUuzmTyyB/27fOffom2WvwVC+N8NcU06MSeeE3pz8AxyT1bVqgaaZgHD6jwfBuSp6oGm7Le1uuucAVTV1PJ//7YpOIw5UY9+sJGQELj97H5uRwkq/haIfDz3Q9wK7G/ivp4EBgIXqGrZMdrNA2aIyCAR6QDcA8xt4j5brbTOMVwxJo1Xl+ewMfeI23GMabFWZB/inbX7mHlqeptaDMgf/lzFdCWe0zyXe7d472t+E5FU4AZgOJArIsXebbqIpHgfpwCo6gfAY8CneGaRzQbua8z+2oqfTuxDXFQ4D7/7jd08Z0wTqCoPvbuBxLhIbji97U3nfTxhfrTJ9v5vKaB1nvtNVbM59qmp2Hrt/wz8ubH7aWvaR0fw04l9efCdDSzanM+Z/ZPcjmRMi/Luun2s2nWYxyYPJSbSn6/DtsWfq5g+w3NJ6rPerZ/3NRMErhiTSlqnaB559xuq7eY5Y/xWXlXDox9sZECXOCa38Sk1GtKYMYjtqjrX+/i/RGRac4cy/osIC+FXkwayZX8xr3xt61Yb46/nl+wk52AZ95w/iNAQuynOF78KhKq+BbwhIo8C7wKISHsReRWbBsN15wxOZmx6J/78780cLq10O44xQW9/UTmPf7KF7w9KZkLfzm7HCVqNmYlqGJ5B5uUiMgNYBxwGRgQimPGfiPDbCwZRWFbFXz7e4nYcY4LeHz7YRGVNLXefN9DtKEHN7wKhqnuBH3p/ZjbwvqreoKolgQpn/DewazzTRqcw/6tstuTZPYXGNGRNzmFeX7Gbayf0Iq1zjNtxgprfBUJEhgOZwHbgIuAsEXlFRNoHKpxpnNu/34+YiFAeeGeDXfZqjA+qyu/ezqJzbCS3nNnH7ThBrzGnmP4D/FlVf+idjXUYnktf1wUkmWm0TrGR/Ox7/fhiSwEff9PU+xmNab0Wrt7Lyl2Huevc/sS10XWmG6MxBWKUqs45+sQ7hfcM4CfNH8s01RVjU+mbFMsD72RRXlXjdhxjgkZReRUPv/cNw3okMGWkXdbqj8aMQficOlRV/9V8ccyJCg8N4XcXDibnYBlPf2azvRpz1N/+s4WC4goeuGgIIXZZq1/a9np6rdS4Pp05f2hXZi3aSs7BUrfjGOO6LXlFPL9kJ5dm9GRYTxs29ZcViFbqnvMHEiLCQ+9ucDuKMa5SVe5/O4voiFDuPKe/23FaFCsQrVTXhHbcOrEPH2blsWiTDVibtuu9dbks2XqAO8/pT6fYSLfjtChWIFqx6yakk54Yw33/sgFr0zYVlVfxwDtZDO4Wz2WnpLodp8WxAtGKRYSF8NBFQ8g+UMqsT7e6HccYx/3535vZX1TBwxefZPMtNYEViFZuXJ/O/HB4N576bDvb8ovdjmOMY9bvKeSFL3cy/ZQUhtvAdJM4ViBE5BYRyRSRChGZe4x2V4tITZ1FhYpF5AyncrZGd58/iMjwEO59a73dYW3ahNpa5Z631tMxJoI7zxngdpwWy8kexF7gIeA5P9ouVdXYOtuiwEZr3RLjIrnr3AF8ue0AC1fvdTuOMQH38te7WJ1zmHvOH0RCO7tjuqkcKxCqusA7bfgBp/Zp/uey0Z5u9oPvbOBQiU0JblqvvCPlPPr+Rsb36cRFw7u5HadFC9YxiBEiUiAim0XkXhHxuRagiMz0nrbKzM/P99XEeIWGCL+ffBKFZZ7pBoxpre5bmEVlTS0P//AkRGxg+kQEY4H4HBgCJAGTgWnAnb4aqupsVc1Q1YzExEQHI7ZMA7rEc8Pp6byxYjdLtha4HceYZvdhVi4fZOVy2/f62lTezSDoCoSqblfVHapaq6rrgAeAKW7nai1uPasvaZ2i+c2b6+zeCNOqFJVXcd/CLAZ0ieP6U9PdjtMqBF2B8EEB6yc2k6jwUB750UlkHyjl/z7e7HYcY5rNox9sJK+onN9PHkp4aEv4agt+Tl7mGiYiUUAoECoiUb7GFkRkkogkex8PAO4FFjqVsy0Y17szU0f15JnPt7Mm57DbcYw5YUu3HeDFr3Zxzbheds9DM3KyzN4DlAG/Ai73Pr5HRFK89zqkeNtNBNaKSAnwHrAAeMTBnG3Cb84fSFJcFHe9sZbK6lq34xjTZKWV1fzyn2tJ7RRtk/E1Mycvc71fVaXedr+q7vLe67DL2+4OVU1W1RhVTVfV36pqlVM524r4qHAevngIm/KKeNym4TAt2J8+2syug6X8/kdDaRcR6nacVsVO1LVhEwcmc/GI7sz6dCsb9h5xO44xjbYi+xDPLdnB5WNSGNu7k9txWh0rEG3cb38wiPbR4dzx+ho71WRalLLKGu58fQ3dEtrxq0kD3Y7TKlmBaOM6xETw8MUnsWHfEf7+yRa34xjjt8c+3Mj2ghIemzKU2Eif99KaE2QFwnDO4C78aGR3Zi3axmq7qsm0AF9uK+D5JTu5amwq4/t0djtOq2UFwgBw3wWDSYqL5BevrbYb6ExQKyqv4s7X15LWKZpfTrKZWgPJCoQBIKFdOI9OHsq2/BL+8OEmt+MY06CH3vmGfYVl/OmSYURH2KmlQLICYf7rtH6JXDEmlTmLd7B4i83VZILPB+tzeTUzhxtO783JqR3djtPqWYEw3/Kb8wbSOzGGX7y+msOlNi24CR77j5Tz6wVrGdI9np9/r5/bcdoEKxDmW9pFhPLXqSM4WFLJb95cZyvQmaBQW6vc8cZayqpq+MulI4gIs68uJ9hRNt8xpHsCt3+/P++ty+WNFbvdjmMMLyzdyeeb87n7/EH0SYp1O06bYQXC+DTztHRO6dWR+/6Vxfb8YrfjmDZsw94j/L/3N3LWgCQuPyXl+D9gmo0VCONTaIjwl6nDiQgL4dZXVlFRbZe+GueVVlZzyysrad8unD9MGWorxDnMCoRpUNeEdvxhyjCy9h7h9+9vdDuOaYPuW5jFjoIS/jJ1OJ1iI92O0+ZYgTDH9P1ByVw9Lo3nl+zk4w15bscxbcjC1Xt4fcVubjmzD+N6293SbnBywaBbRCRTRCpEZO5x2v5cRHJFpFBEnhMR+6eDi3593gAGdY3njjfWsPtQqdtxTBuwLb+Y3yxYR0ZqB26b2NftOG2Wkz2IvcBDwHPHaiQi5+BZVGgikAakA78LdDjTsMiwUJ6YPpLqGuWWl1fZrK8moMoqa7j5xZVEhIXwt2kjCLPlQ13j5IJBC1T1LeDAcZpeBcxR1SxVPQQ8CFwd6Hzm2Hp1juEPU4ayOucwj7z3jdtxTCt278L1bN5fxF+mjqBb+3Zux2nTgrE0DwbW1Hm+BkgWEVsNxGWTTurKNePTmPvlTt5du8/tOKYVei0zhzdW7ObWM/twer9Et+O0ecFYIGKBwjrPjz6Oq99QRGZ6xzUy8/PzHQnX1v160kBGpLTnrjfWsHV/kdtxTCuyfk8h9761nnG9O3GbTaURFIKxQBQD8XWeH338nW8jVZ2tqhmqmpGYaP/acEJEWAizpo+kXUQoM+evoKjclgs3J+5gSSU3zF9Bx5gI/jZtBKEhdr9DMAjGApEFDKvzfBiQp6rHG7swDuma0I7HLxtJ9oFSbn9tDbW1Nl+TabrqmlpufWUl+cUVPHX5yXS2+x2ChpOXuYaJSBQQCoSKSJSI+JrMfR4wQ0QGiUgH4B5grlM5jX/GpHfi7vMG8u8NeTz+6Va345gW7A8fbgGUCaEAABIkSURBVGLJ1gM89MMhDOvZ3u04pg4nexD3AGV4LmG93Pv4HhFJEZFiEUkBUNUPgMeAT4Fs73afgzmNn64Zn8bFI7rzfx9v5qOsXLfjmBborVV7ePrz7Vw+JoVLMnq6HcfUI61lOueMjAzNzMx0O0abU15Vw6VPL2XL/mL+edM4BnaNP/4PGQOs2nWIS2d/xciU9syfcQrhdr+DK0Rkhapm+HrP/h8xJyQqPJTZV2YQFxXGdS9kUlBc4XYk0wLsKyxj5vwVdImP4snpJ1txCFL2/4o5YcnxUTxzZQYHSiq4cf4Km/nVHFNpZTXXz8ukrLKGZ6/KoENMhNuRTAOsQJhmMbRHe/7442FkZh/irjfW2kp0xqeaWuWnr6xmw94j/G3acPolf+f2JhNEfF1FZEyT/GBoN3YdLOWxDzaR0jGaX5zd3+1IJsg8+M4GPv4mjwcuGsxZA5LdjmOOwwqEaVY3nd6bXQdK+fsnW+nZIZpLRtmVKcbjucU7mPvlTmZM6MWVY9PcjmP8YAXCNCsR4cEfDmHP4TJ+8+Y6kuIjOaN/ktuxjMveX7ePB9/dwDmDk/nNeQPdjmP8ZGMQptmFh3qm4+jfJY6bXlzJql2H3I5kXPTltgJu+8dqRqZ04C+X2jQaLYkVCBMQcVHhzL1mNEnxkVw7dzlb9xe7Hcm4YP2eQmbOW0Fqp2jmXJVBu4hQtyOZRrACYQImMS6SedeOJjREuOq5r9l7uMztSMZB2QdKuPr55cRHhTFvxmjaR9vlrC2NFQgTUKmdYph7zWiOlFVx+bPLyC+yG+nagr2Hy7jsmWXU1NbywrWj6ZpgC/+0RFYgTMAN6Z7A89eMYl9hOVfMWcbh0kq3I5kA2l9UzvRnl3GkrIr5M06hr93r0GJZgTCOyEjryDNXZrA9v4Srnvva1pFopQ6VVHLlnK/JLSxn7rWjGNI9we1I5gRYgTCOmdC3M7OmjyRr7xGutCLR6hwqqWT6s8vYXlDCs1dlcHJqR7cjmRNkBcI46nuDknn8spGs213Ilc99zRErEq3CwZJKLnt2GVvzi3n2ygzG9+nsdiTTDKxAGMedO6QLT0z3Fok5ViRauoNHew7e4nBaP1v+t7VwckW5jiLypoiUiEi2iFzWQLv7RaTKu4jQ0S3dqZzGGecM7uI93VTIZc98xQGbJrxFyjtSzqVPL2V7fjHPWHFodZzsQTwBVALJwHTgSREZ3EDbV1U1ts623bGUxjFnD+7C7Csz2JJXzKWzvyLvSLnbkUwj5Bws5cdPLWXv4TLmXjPaikMr5EiBEJEYYDJwr6oWq+pi4F/AFU7s3wSvM/sn8cK1o8ktLGfKU1+y60Cp25GMH7buL+LHTy2lsKyKl64fw9jendyOZALAqR5EP6BGVTfXeW0N0FAP4gIROSgiWSJyU0MfKiIzRSRTRDLz8/ObM69x0Jj0Trx03SkUlVfzoyeXsG53oduRzDEs33mQyU8upbpWefWGMQzv2d7tSCZAnCoQsUD9v/pCwNcdNK8BA4FE4HrgtyIyzdeHqupsVc1Q1YzEROvetmTDerbnjRvHERkWyqWzl/LZZiv4weiD9blc/uwyOsVE8ObN4xjQxdYgb82cKhDFQP3fpHigqH5DVd2gqntVtUZVvwT+CkxxIKNxWZ+kWBbcPI7UTjHMmLuc15bnuB3JeKkqc5fs4KaXVjCoWzxv3DSOnh2j3Y5lAsypArEZCBORvnVeGwZk+fGzCtj8wG1EcnwUr93gOad91z/X8sh731BTa8uXuqmqppZ7F67n/rc3MHFAMi9fN4aOto50m+BIgVDVEmAB8ICIxIjIeOAiYH79tiJykYh0EI/RwE+BhU7kNMEhLiqc564exRVjUpn9+XZumL+C4opqt2O1SYVlVVw7dzkvfrWLG05L5+krTrYpu9sQJy9zvRloB+wHXgFuUtUsETlVROouFjAV2Irn9NM84FFVfcHBnCYIhIeG8OAPh/C7CwfzycY8fjRrCdvzbU0JJ23KLeKixxfz1fYDPDZlKL8+b6At9tPGiGrr6L5nZGRoZmam2zFMACzeUsCtr6ykukb586XD+f4gW+w+0N5es5e73lhLbFQYs6aPZFSazavUWonIClXN8PWeTbVhgt6Evp15+9YJpHaO5vp5mfzhw41U19S6HatVqqiu4XdvZ3HrK6sY3C2ed2+dYMWhDbMCYVqEHh2ieePGcVya0ZMnPt3G1NlfscdWqGtWOwtKmPLkUp5fspOrx6Xx8vVjSIqPcjuWcZEVCNNiRIWH8uiUofx16nA25hZx3l+/4IP1+9yO1eKpKm+t2sMP/r6YXQdLefqKk7n/wsFEhNnXQ1tnvwGmxbloeHfe/ekEUjtFc+OLK/n5q6spLLMZYZviQHEFN7+0kp+9upoBXeJ477ZTOWdwF7djmSAR5nYAY5oitVMM/7xpHI9/spXHP93K0m0HeHTKUE63CeP89mFWLne/uY4jZdX88twBzDwt3a5SMt9iPQjTYoWHhvDz7/fjzZvHERsVxlXPfc1t/1hFgU0dfky5heXcOH8FN8xfQWJcFP+6dTw3ndHbioP5DrvM1bQK5VU1zFq0jScXbSU6IoxfTxrAJRk9CbEvvf+qrqnlpWW7+MOHm6iqqeW27/Xl+lPTCQ+1fye2Zce6zNUKhGlVtuQV8Zs317F85yFO6p7AfRcMIsMu02TJ1gIeeHsDm/KKmNCnMw9fPITUTjFuxzJBwAqEaVNUlYWr9/L79zeSe6ScHwztyh1n9yetc9v7QtySV8QfPtzERxvy6NGhHXefN5Bzh3RBxHpWxsMKhGmTSiureWrRNp75YgeVNbVcktGT2yb2pUtC67+2P+dgKX/5eAtvrtpNdEQYN56eznWnphMVbvMomW+zAmHatP1F5TzxyVZe/noXIsKUk3tww2nprfIUy9b9xTz12TbeWrWHkBDhqrGp3HRGH5t91TTICoQxeP5VPWvRNv65YjfVtbWcd1JXrhnfi5Ep7Vv0KRdVZdmOg8xdspMPN+QSGRbC1FEp3HB6Ol0T2rkdzwQ5KxDG1LH/SDlzFu/g5WW7KKqoZnC3eK4cm8r5Q7sRG9lybg06Ul7Fv1bvZf7SbDblFZHQLpwrxqRyzfg0OsVGuh3PtBBWIIzxoaSimjdX7WHe0p1sziumXXgo5w7pwsUjujO2d6egvPyzsrqWxVvzWbByDx9tyKOyupZBXeO5elwaFwzrZms1mEazAmHMMagqK7IPsWDVHt5Zs5cj5dUktAtn4sAkzh7UhfF9OhEXFe5avsLSKhZvLeDDrFw+3bifoopqOkSHc+Gwblw8sgfDeiS06FNkxl1BUSBEpCMwBzgbKAB+raov+2gnwO+B67wvzQF+qccJagXCNIfyqhoWbcrnow25/Oeb/RSWVREaIgzrkcD4Pp0ZmdqBYT3aB3TQt6C4gjU5h1mRfYglWwtYt6eQWoWOMRF8z1u0TuuXaJPpmWZxrALh5AnXJ4BKIBkYDrwrImtUtf661DOBH+JZs1qBfwPbgacczGraqCjvaaZzh3ShqqaWzJ2eL+kl2wqYtWjbf9fH7tmxHf2T4+mTFEvvxBh6dIimS0IUXeKj/DrNU1JRTe6RcvIKy9l9qIxt+cVsyy/mm31F/53GPCxEGJHSnlvP6suEvp0Z0bM9YUF42su0Xo70IEQkBjgEDFHVzd7X5gN7VPVX9dp+CcxV1dne5zOA61V1zLH2YT0IE2jFFdWs31PImpzDrNl9mC15xew8UEJVzbf/hiLDQoiLCiMmMowI7xe64hk/KKmopqiimsrqby94FBEWQnrnGPokxTKsR3uGp7RncLd4oiNazqC5aZmCoQfRD6g5Why81gCn+2g72Pte3XaDfX2oiMzE0+MgJSWleZIa04DYyDDGpHdiTHqn/75WXVNLzqEy9h4uI7ewnNwj5Rwpq6Koopri8mqqa/9XCMJDQ4iNDCM2Koz27SLokhBJcnwU3du3o0eHaJsszwQdpwpELFBY77VCIM6PtoVArIhI/XEIby9jNnh6EM0X1xj/hIWG0KtzDL3a4DQepvVz6oRmMRBf77V4oMiPtvFA8fEGqY0xxjQvpwrEZiBMRPrWeW0YUH+AGu9rw/xoZ4wxJoAcKRCqWgIsAB4QkRgRGQ9cBMz30XwecLuIdBeRbsAvgLlO5DTGGPM/Tl4zdzPQDtgPvALcpKpZInKqiBTXafc08DawDlgPvOt9zRhjjIMcu4ZOVQ/iub+h/utf4BmYPvpcgbu8mzHGGJfYXTfGGGN8sgJhjDHGJysQxhhjfGo1s7mKSD6Q3cQf74xnAsFgY7kax3I1XrBms1yNcyK5UlU10dcbraZAnAgRyWxoLhI3Wa7GsVyNF6zZLFfjBCqXnWIyxhjjkxUIY4wxPlmB8JjtdoAGWK7GsVyNF6zZLFfjBCSXjUEYY4zxyXoQxhhjfLICYYwxxicrEMYYY3xqcwVCRCJFZI6IZItIkYisEpFJx/mZn4tIrogUishzIhIZoGy3iEimiFSIyNzjtL1aRGpEpLjOdobbubztnTpeHUXkTREp8f7/edkx2t4vIlX1jle601nE41EROeDdHhORgK012ohcAT0+9fbVmN9zR36XGpvN4b+/Rn1nNecxa3MFAs8Mtjl41sNOAO4FXhORNF+NReQc4FfARCANSAd+F6Bse4GHgOf8bL9UVWPrbIvczuXw8XoCqASSgenAkyLic/1yr1frHa/tLmSZiWdW42HAUOAHwA3NmKOpuSCwx6cuv36fHP5dalQ2L6f+/vz+zmr2Y6aqbX4D1gKTG3jvZeCROs8nArkBzvMQMPc4ba4GFjt8nPzJ5cjxAmLwfPH1q/PafOD3DbS/H3gxQMfF7yzAl8DMOs9nAF8FQa6AHZ+m/j658bfXiGyO//3V27/P76zmPmZtsQfxLSKSDPSj4WVNBwNr6jxfAySLSKdAZ/PDCBEpEJHNInKviDi2vscxOHW8+gE1qrq53r6O1YO4QEQOikiWiNzkUhZfx+dYmZ3KBYE7Pk0VzH974NLf33G+s5r1mLXpAiEi4cBLwAuqurGBZrFAYZ3nRx/HBTKbHz4HhgBJwGRgGnCnq4k8nDpe9fdzdF8N7ec1YCCQCFwP/FZEprmQxdfxiQ3QOERjcgXy+DRVsP7tgUt/f358ZzXrMWt1BUJEFomINrAtrtMuBE93uxK45RgfWQzE13l+9HFRIHL5S1W3q+oOVa1V1XXAA8CUxn5Oc+fCueNVfz9H9+VzP6q6QVX3qmqNqn4J/JUmHK8GNCaLr+NTrN7zAc3M71wBPj5N1Sy/S4HQXH9/jeHnd1azHrNWVyBU9QxVlQa2CeC5kgSYg2fgbrKqVh3jI7PwDCgeNQzIU9UDzZ3rBCnQ6H+FBiCXU8drMxAmIn3r7auhU4Xf2QVNOF4NaEwWX8fH38yBzFVfcx6fpmqW3yWHBPR4NeI7q1mPWasrEH56Ek93+gJVLTtO23nADBEZJCIdgHuAuYEIJSJhIhIFhAKhIhLV0HlNEZnkPReJiAzAc2XDQrdz4dDxUtUSYAHwgIjEiMh44CI8/8Ly9d9wkYh0EI/RwE9ppuPVyCzzgNtFpLuIdAN+QYB+nxqTK5DHx8e+/P19cuxvr7HZnPz78/L3O6t5j5lbo/BubUAqnmpfjqc7dnSb7n0/xfs8pc7P3A7kAUeA54HIAGW735ut7na/r1zAH72ZSoDteLq44W7ncvh4dQTe8h6DXcBldd47Fc+pm6PPXwEOeLNuBH7qRBYfOQR4DDjo3R7DOyeak8fI6ePjz++Tm79Ljc3m8N9fg99ZgT5mNlmfMcYYn9rqKSZjjDHHYQXCGGOMT1YgjDHG+GQFwhhjjE9WIIwxxvhkBcIYY4xPViCMMcb4ZAXCGGOMT1YgjDHG+GQFwpgAEJHe3rUVRnqfd/OuHXCGy9GM8ZtNtWFMgIjI9XjmxTkZeBNYp6p3uJvKGP9ZgTAmgETkX0AvPJOtjVLVCpcjGeM3O8VkTGA9g2flsb9bcTAtjfUgjAkQEYnFsybwp8Ak4CRVPehuKmP8ZwXCmAARkTlAnKpeIiKzgfaqeonbuYzxl51iMiYAROQi4FzgRu9LtwMjRWS6e6mMaRzrQRhjjPHJehDGGGN8sgJhjDHGJysQxhhjfLICYYwxxicrEMYYY3yyAmGMMcYnKxDGGGN8sgJhjDHGp/8PWA5XvPpRWP4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_function(f, 'x', 'x**2')\n",
    "plt.scatter(-1.5, f(-1.5), color='red');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculating Gradients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xt = tensor(3.).requires_grad_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(9., grad_fn=<PowBackward0>)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yt = f(xt)\n",
    "yt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yt.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(6.)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xt.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 3.,  4., 10.], requires_grad=True)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xt = tensor([3.,4.,10.]).requires_grad_()\n",
    "xt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(125., grad_fn=<SumBackward0>)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def f(x): return (x**2).sum()\n",
    "\n",
    "yt = f(xt)\n",
    "yt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 6.,  8., 20.])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yt.backward()\n",
    "xt.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stepping With a Learning Rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### An End-to-End SGD Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19.])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "time = torch.arange(0,20).float(); time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAAD7CAYAAACYLnSTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAWy0lEQVR4nO3dfYxcV3nH8e8vtpWsbC+u48XFW9luDLGpExw3i4KIAkhJa0FLcWMkTFIISMhAlKqo1IK0OLh5UQBD/yiEF0sp5LUNhrUhpGA1SlJICikbXMdaYVt1UkPWKawhXrz2OjHu0z/mTjKezM7c8cydlzu/jzSS59wzd56czD5z5txzz1FEYGZm3e2sdgdgZmaNczI3M8sBJ3MzsxxwMjczywEnczOzHJjZjjddsGBBLF26tB1vbWbWtZ544onDETFQ6VhbkvnSpUsZGRlpx1ubmXUtSQenO+ZhFjOzHHAyNzPLASdzM7MccDI3M8sBJ3Mzsxxoy2yWM7Vj1xhbdu7j0JEpFs3rY+Oa5axdPdjusMzM2q5rkvmOXWNcP7yHqZOnABg7MsX1w3sAnNDNrOd1zTDLlp37XkzkRVMnT7Fl5742RWRm1jm6JpkfOjJVV7mZWS/pmmS+aF5fXeVmZr2ka5L5xjXL6Zs147Syvlkz2LhmeZsiMjPrHF1zAbR4kdOzWczMXq5rkjkUErqTt5nZy3XNMIuZmU3PydzMLAeczM3McsDJ3MwsB2omc0mTZY9Tkj5fcvxySXslHZf0sKQl2YZsZmblaibziJhTfAALgSlgG4CkBcAwsAmYD4wA92UXrpmZVVLvMMs7gV8CP0ieXwmMRsS2iDgBbAZWSVrRvBDNzKyWepP5NcCdERHJ85XA7uLBiDgGHEjKTyNpg6QRSSPj4+NnGq+ZmVWQOplLWgy8GbijpHgOMFFWdQKYW/76iNgaEUMRMTQwMHAmsZqZ2TTq6Zm/F3g0Ip4uKZsE+svq9QNHGw3MzMzSqzeZ31FWNgqsKj6RNBtYlpSbmVmLpErmkt4IDJLMYimxHbhA0jpJ5wA3AE9GxN7mhmlmZtWkXWjrGmA4Ik4bPomIcUnrgC8AdwOPA+ubG6KZWffLeg/jVMk8Ij5Y5diDgKcimplNoxV7GPt2fjOzjLViD2MnczOzjLViD2MnczOzjLViD2MnczOzjLViD+Ou2jbOzKwbtWIPYydzM7MWyHoPYw+zmJnlgJO5mVkOOJmbmeWAk7mZWQ44mZuZ5YCTuZlZDjiZm5nlgJO5mVkOOJmbmeWAk7mZWQ6kTuaS1kv6qaRjkg5Iuiwpv1zSXknHJT0saUl24ZqZWSWp1maR9EfAp4F3Af8JvCopXwAMAx8A7gduAu4D3pBFsI3KetsmM7N2SbvQ1t8DN0bEj5LnYwCSNgCjEbEteb4ZOCxpRadt6tyKbZvMzNql5jCLpBnAEDAg6b8lPSPpC5L6gJXA7mLdiDgGHEjKy8+zQdKIpJHx8fHm/Rek1Iptm8zM2iXNmPlCYBbwTuAy4CJgNfAJYA4wUVZ/AphbfpKI2BoRQxExNDAw0FDQZ6IV2zaZmbVLmmRezHafj4hnI+Iw8A/A24BJoL+sfj9wtHkhNkcrtm0yM2uXmsk8Ip4DngGiwuFRYFXxiaTZwLKkvKO0YtsmM7N2SXsB9KvAX0r6HnAS+AjwHWA7sEXSOuAB4AbgyU67+Amt2bbJzPKr02fDpU3mNwELgP3ACeDrwC0RcSJJ5F8A7gYeB9ZnEWgzZL1tk5nlUzfMhkuVzCPiJHBt8ig/9iCwoslxmZl1jGqz4Tolmft2fjOzGrphNpyTuZlZDd0wG87J3Myshm6YDZf2AqiZWc/qhtlwTuZmZil0+mw4D7OYmeWAk7mZWQ44mZuZ5YCTuZlZDjiZm5nlgJO5mVkOOJmbmeWAk7mZWQ44mZuZ5YCTuZlZDjiZm5nlQKpkLukRSSckTSaPfSXHrpJ0UNIxSTskzc8uXDMzq6Senvl1ETEneSwHkLQS+ArwHmAhcBz4YvPDNDOzahpdNfFq4P6I+D6ApE3ATyXNjYijDUdnZmap1NMzv1XSYUmPSXpLUrYS2F2sEBEHgBeA88tfLGmDpBFJI+Pj443EbGZmZdIm848B5wGDwFbgfknLgDnARFndCWBu+QkiYmtEDEXE0MDAQAMhm5lZuVTJPCIej4ijEfF8RNwBPAa8DZgE+suq9wMeYjEza6EznZoYgIBRYFWxUNJ5wNnA/sZDMzOztGpeAJU0D7gE+Hfgt8C7gDcBH0le/0NJlwE/AW4Ehn3x08ystdLMZpkF3AysAE4Be4G1EbEPQNKHgHuAc4EHgfdnE6qZmU2nZjKPiHHg9VWO3wvc28ygzMysPr6d38wsBxq9aain7Ng1xpad+zh0ZIpF8/rYuGY5a1cPtjssMzMn87R27Brj+uE9TJ08BcDYkSmuH94D4IRuZm3nYZaUtuzc92IiL5o6eYotO/dN8wozs9ZxMk/p0JGpusrNzFrJyTylRfP66io3M2slJ/OUNq5ZTt+sGaeV9c2awcY1y9sUkZnZS3wBNKXiRU7PZjGzTuRkXoe1qwedvM2sI3mYxcwsB5zMzcxywMMsZtYT8n4Ht5O5meVeL9zB7WEWM8u9XriD28nczHKvF+7gdjI3s9zrhTu4nczNLPd64Q7uupK5pNdIOiHp7pKyqyQdlHRM0g5J85sfppnZmVu7epBbr7yQwXl9CBic18etV16Ym4ufUP9sltuAHxefSFoJfAX4EwobOm8Fvgisb1aAZmbNkPc7uFMnc0nrgSPAfwCvToqvBu6PiO8ndTYBP5U0NyKONjtYMzOrLNUwi6R+4Ebgo2WHVgK7i08i4gDwAnB+hXNskDQiaWR8fPzMIzYzs5dJO2Z+E3B7RPy8rHwOMFFWNgHMLT9BRGyNiKGIGBoYGKg/UjMzm1bNYRZJFwFXAKsrHJ4E+svK+gEPsZiZtVCaMfO3AEuBn0mCQm98hqQ/AL4HrCpWlHQecDawv9mBmpnZ9NIk863Av5Q8/xsKyf3DwCuBH0q6jMJslhuBYV/8NDNrrZrJPCKOA8eLzyVNAiciYhwYl/Qh4B7gXOBB4P0ZxWpmZtOoe9XEiNhc9vxe4N5mBWRmZvXz7fxmZjngZG5mlgNO5mZmOeBkbmaWA07mZmY54GRuZpYDTuZmZjngZG5mlgNO5mZmOeBkbmaWA07mZmY5UPfaLGZm7bBj1xhbdu7j0JEpFs3rY+Oa5bne07NeTuZm1vF27Brj+uE9TJ08BcDYkSmuH94D4ISe8DCLmXW8LTv3vZjIi6ZOnmLLzn1tiqjzOJmbWcc7dGSqrvJe5GRuZh1v0by+usp7UapkLuluSc9K+o2k/ZI+UHLsckl7JR2X9LCkJdmFa2a9aOOa5fTNmnFaWd+sGWxcs7xNEXWetD3zW4GlEdEP/Blws6SLJS0AhoFNwHxgBLgvk0jNrGetXT3IrVdeyOC8PgQMzuvj1isv9MXPEqlms0TEaOnT5LEMuBgYjYhtAJI2A4clrYiIvU2O1cx62NrVg07eVaQeM5f0RUnHgb3As8C/AiuB3cU6EXEMOJCUl79+g6QRSSPj4+MNB25mZi9Jncwj4lpgLnAZhaGV54E5wERZ1YmkXvnrt0bEUEQMDQwMnHnEZmb2MnXNZomIUxHxKPB7wIeBSaC/rFo/cLQ54ZmZWRpnOjVxJoUx81FgVbFQ0uyScjMza5GayVzSKyWtlzRH0gxJa4B3Aw8B24ELJK2TdA5wA/CkL36ambVWmp55UBhSeQZ4Dvgs8JGI+FZEjAPrgFuSY5cA6zOK1czMplFzamKSsN9c5fiDwIpmBpVXXvXNzLLiVRNbxKu+Wa9zZyZbXpulRbzqm/WyYmdm7MgUwUudmR27xtodWm44mbeIV32zXubOTPaczFvEq75ZL3NnJntO5i3iVd+sl7kzkz0n8xbxqm/Wy9yZyZ5ns7SQV32zXlX83Hs2S3aczM2sJdyZyZaHWczMcsDJ3MwsB5zMzcxywMnczCwHfAG0i3htCzObjpN5l/BCXWZWjYdZuoTXtjCzapzMu4TXtjCzatJsG3e2pNslHZR0VNIuSW8tOX65pL2Sjkt6WNKSbEPuTV7bwsyqSdMznwn8nMJuQ68ANgFfl7RU0gJgOCmbD4wA92UUa0/z2hZmVk2abeOOAZtLir4j6WngYuBcYDQitgFI2gwclrTCmzo3VzPWtvBsGLP8qns2i6SFwPnAKIWNnncXj0XEMUkHgJXA3rLXbQA2ACxevLiBkHtXI2tbeDaMWb7VdQFU0izgHuCOpOc9B5goqzYBzC1/bURsjYihiBgaGBg403jtDHk2jFm+pU7mks4C7gJeAK5LiieB/rKq/cDRpkRnTePZMGb5liqZSxJwO7AQWBcRJ5NDo8CqknqzgWVJuXUQz4Yxy7e0PfMvAa8F3h4RpV257cAFktZJOge4AXjSFz87j2fDmOVbmnnmS4APAhcB/ytpMnlcHRHjwDrgFuA54BJgfZYB25nxtnVm+aaIaPmbDg0NxcjISMvf18ysm0l6IiKGKh3z7fxmZjngZG5mlgNeAtfMUvEdxJ3NydzMavIdxJ3PwyxmVpPvIO58TuZmVpPvIO58TuZmVpPvIO58TuZmVpPvIO58vgBqZjU1Yz19y5aTuZml0sh6+pY9J3NLzfOMzTqXk7ml4nnGZp3NF0AtFc8zNutsTuaWiucZm3U2D7NYKovm9TFWIXHXM8/YY+5m2XHP3FJpdJ5xccx97MgUwUtj7jt2jWUQrVnvSbsH6HWSRiQ9L+lrZccul7RX0nFJDyc7E1nONLpTkcfc22/HrjEu/dRD/P7HH+DSTz3kL9KcSTvMcgi4GVgDvPi7WtICYBj4AHA/cBNwH/CG5oZpnaCRecYec28vz0bKv1Q984gYjogdwK/KDl0JjEbEtog4AWwGVkla0dwwrdt5bY/28i+j/Gt0zHwlsLv4JCKOAQeS8tNI2pAM1YyMj483+LbWbby2R3v5l1H+NZrM5wATZWUTwNzyihGxNSKGImJoYGCgwbe1btPomLs1xr+M8q/RqYmTQH9ZWT9wtMHzWg55bY/22bhm+Wlj5uBfRnnTaM98FFhVfCJpNrAsKTezDuFfRvmXqmcuaWZSdwYwQ9I5wG+B7cAWSeuAB4AbgCcjYm9G8ZrZGfIvo3xL2zP/BDAFfBz4i+Tfn4iIcWAdcAvwHHAJsD6DOM3MrIpUPfOI2Exh2mGlYw8CnopoZtZGvp3fzCwHnMzNzHLAydzMLAe8BK5Zl/ASwlaNk7lZF/BCWVaLh1nMuoAXyrJanMzNuoAXyrJaPMxiXaOXx4ybsW2f5Zt75tYV8rDtXCM7/XgJYavFydy6QrePGTf6ZeSFsqwWD7NYV+j2MeNqX0ZpE7IXyrJq3DO3rtDtmyt0+5eRdT4nc+sK3T5m3O1fRtb5nMytK3T7mHG3fxlZ5/OYuXWNbh4zLsbdq1MrLXtO5mYt0s1fRtb5mjLMImm+pO2Sjkk6KOmqZpzXzMzSaVbP/DbgBWAhcBHwgKTdEeGNnS03evkOVOt8DffMJc2msA/opoiYjIhHgW8D72n03GadIg93oFq+NWOY5XzgVETsLynbDaxswrnNmqaR2+m7/Q5Uy79mDLPMASbKyiaAuaUFkjYAGwAWL17chLc1S6/R9cB90491umb0zCeB/rKyfuBoaUFEbI2IoYgYGhgYaMLbmqXXaM/aN/1Yp2tGMt8PzJT0mpKyVYAvflrHaLRn7Zt+rNM1nMwj4hgwDNwoabakS4F3AHc1em6zZmm0Z93td6Ba/jVrauK1wD8BvwR+BXzY0xKtk2xcs/y0MXOov2ftm36skzUlmUfEr4G1zTiXWRZ8O73lnW/nt57hnrXlmVdNNDPLASdzM7MccDI3M8sBJ3MzsxxwMjczywFFROvfVBoHDjZwigXA4SaFkwXH1xjH1xjH15hOjm9JRFRcD6UtybxRkkYiYqjdcUzH8TXG8TXG8TWm0+ObjodZzMxywMnczCwHujWZb213ADU4vsY4vsY4vsZ0enwVdeWYuZmZna5be+ZmZlbCydzMLAeczM3McqAjk7mk+ZK2Szom6aCkq6apJ0mflvSr5PEZSco4trMl3Z7EdVTSLklvnabu+ySdkjRZ8nhLlvEl7/uIpBMl71lxo8s2td9k2eOUpM9PU7cl7SfpOkkjkp6X9LWyY5dL2ivpuKSHJS2pcp6lSZ3jyWuuyDI+SW+Q9G+Sfi1pXNI2Sa+qcp5Un4smxrdUUpT9/9tU5Tytbr+ry2I7nsR78TTnyaT9mqUjkzlwG/ACsBC4GviSpJUV6m2gsCnGKuB1wJ8CH8w4tpnAz4E3A68ANgFfl7R0mvo/jIg5JY9HMo6v6LqS95xuO52Wt19pW1D4/zsFbKvykla03yHgZgq7Zb1I0gIKWyJuAuYDI8B9Vc7zz8Au4Fzg74BvSGrG7uUV4wN+h8LMi6XAEgqbqH+1xrnSfC6aFV/RvJL3vKnKeVrafhFxT9nn8VrgKeAnVc6VRfs1Rcclc0mzgXXApoiYjIhHgW8D76lQ/RrgcxHxTESMAZ8D3pdlfBFxLCI2R8T/RMT/RcR3gKeBit/mHa7l7VfmnRS2GvxBC9/zZSJiOCJ2UNjysNSVwGhEbIuIE8BmYJWkFeXnkHQ+8IfAJyNiKiK+Ceyh8FnOJL6I+G4S228i4jjwBeDSRt+vWfHVox3tV8E1wJ3RpVP8Oi6ZA+cDpyJif0nZbqBSz3xlcqxWvcxIWkgh5un2PF0t6bCk/ZI2SWrV7k63Ju/7WJWhiXa3X5o/nna1H5S1T7J5+QGm/yw+FRFHS8pa3Z5vYvrPYVGaz0WzHZT0jKSvJr92Kmlr+yXDZ28C7qxRtR3tl0onJvM5wERZ2QQwN0XdCWBO1uO+RZJmAfcAd0TE3gpVvg9cALySQg/j3cDGFoT2MeA8YJDCz/D7JS2rUK9t7SdpMYWhqjuqVGtX+xU18lmsVrfpJL0OuIHq7ZP2c9Esh4HXUxgCuphCW9wzTd22th/wXuAHEfF0lTqtbr+6dGIynwT6y8r6KYwH1qrbD0y24meSpLOAuyiM7V9XqU5EPBURTyfDMXuAGykMLWQqIh6PiKMR8XxE3AE8BrytQtW2tR+FP55Hq/3xtKv9SjTyWaxWt6kkvRr4LvBXETHtkFUdn4umSIZJRyLitxHxCwp/J38sqbydoI3tl3gv1TsWLW+/enViMt8PzJT0mpKyVVT++TiaHKtVr6mSnuvtFC7grYuIkylfGkBLfjWkfN+2tF+i5h9PBa1uv9PaJ7mes4zpP4vnSSrtSWbensnwwIPATRFxV50vb3V7FjsJ030WW95+AJIuBRYB36jzpe36e66o45J5Mi45DNwoaXbS0O+g0Asudyfw15IGJS0CPgp8rQVhfgl4LfD2iJiarpKktyZj6iQXzTYB38oyMEnzJK2RdI6kmZKupjAWuLNC9ba0n6Q3UvipWm0WS8vaL2mnc4AZwIxi2wHbgQskrUuO3wA8WWlILbnG81/AJ5PX/zmFGULfzCo+SYPAQ8BtEfHlGueo53PRrPgukbRc0lmSzgX+EXgkIsqHU9rSfiVVrgG+WTZeX36OzNqvaSKi4x4UpoHtAI4BPwOuSsovozAMUKwn4DPAr5PHZ0jWm8kwtiUUvpFPUPhpWHxcDSxO/r04qftZ4BfJf8dTFIYJZmUc3wDwYwo/T48APwL+qFPaL3nfrwB3VShvS/tRmKUSZY/NybErgL0UplA+Aiwted2XgS+XPF+a1JkC9gFXZBkf8Mnk36Wfw9L/v38LfLfW5yLD+N5NYabXMeBZCp2H3+2U9kuOnZO0x+UVXteS9mvWwwttmZnlQMcNs5iZWf2czM3McsDJ3MwsB5zMzcxywMnczCwHnMzNzHLAydzMLAeczM3McuD/AdndnL7Vn+NhAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "speed = torch.randn(20)*3 + 0.75*(time-9.5)**2 + 1\n",
    "plt.scatter(time,speed);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(t, params):\n",
    "    a,b,c = params\n",
    "    return a*(t**2) + (b*t) + c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mse(preds, targets): return ((preds-targets)**2).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 1: Initialize the parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = torch.randn(3).requires_grad_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "orig_params = params.clone()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 2: Calculate the predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = f(time, params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_preds(preds, ax=None):\n",
    "    if ax is None: ax=plt.subplots()[1]\n",
    "    ax.scatter(time, speed)\n",
    "    ax.scatter(time, to_np(preds), color='red')\n",
    "    ax.set_ylim(-300,100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAEACAYAAABLfPrqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAc1ElEQVR4nO3dfbBU9Z3n8fcHsYAAd8B4hyxMAaurMHV1iOtNOZuYxIxuTDKbLUf2D81NHGY2koxFaqusMjErRCtKaeLU/pHZyQOWBHWZncQUWuM86M5UxESzk9pmLExuBHeJXBKfckkQuYD4wHf/OKe1bbvvvX3POd2nuz+vqi66z+/Xp78cDv3t83s6igjMzMxaMavTAZiZWfdx8jAzs5Y5eZiZWcucPMzMrGVOHmZm1jInDzMza5mTh5mZtSzX5CFpg6SKpBOSttWVXSxpj6Rjkh6WtKKmbI6krZJekvS8pGvzjMvMzPKV95XHs8AtwNbajZJOB3YAm4DTgArwnZoqNwFnASuADwGfl/SRnGMzM7OcqIgZ5pJuAX4nItalr9cD6yLivenr+cBB4LyI2CPpGeBPIuJ/peU3A2dFxBW5B2dmZpnNbtPnDAG7qy8i4qikfcCQpBeApbXl6fPLGu0oTUTrAebPn3/+6tWrCwvazKwX7dq162BEDGbZR7uSxwJgvG7bYWBhWlZ9XV/2NhGxBdgCMDw8HJVKJd9Izcx6nKSxrPto12irCWCgbtsAcCQto668WmZmZiXUruQxCqypvkj7PM4ERiPiEPBcbXn6fLRNsZmZWYvyHqo7W9Jc4BTgFElzJc0G7gPOkbQ2Lf8S8ERE7EnfejewUdJiSauBq4FtecZmZmb5yfvKYyNwHLge+GT6fGNEjANrgc3AIeACoHYk1Y3APmAMeAS4PSIezDk2MzPLSSFDddvFHeZmZq2TtCsihrPsw8uTmJlZy5w8zMysZU4eZmbWMicPMzNrWbtmmJfS/Y8/w+0P7eXZF4+zdNE8rrt0FZedt6zTYZmZlV7fJo/7H3+GL+74CcdffR2AZ148zhd3/ATACcTMbAp922x1+0N730gcVcdffZ3bH9rboYjMzLpH3yaPZ1883tJ2MzN7U982Wy1dNI9nGiSKpYvmdSAaM7PGyto327dXHtdduop5p57ylm3zTj2F6y5d1aGIzMzeqto3+8yLxwne7Ju9//FnOh1a/yaPy85bxq2Xn8uyRfMQsGzRPG69/NxSZHQzMyh332zfNltBkkCcLMysrMrcN9vXySOrsrZFmllvKHPfbN82W2VV5rZIMyuX+x9/hvfd9n3+9fV/x/tu+/60vyfK3Dfr5DFDZW6LNLPyyPJDs8x9s262mqEyt0WaWXlM9kNzOkmgrH2zvvKYoWZtjmVoizSz8ujVH5ptTR6Sdkp6WdJE+thbU/YJSWOSjkq6X9Jp7YytVWVuizSz8ujVH5qduPLYEBEL0scqAElDwLeATwFLgGPA1zsQ27SVuS3SzMqjV39olqXPYwR4ICJ+ACBpE/CkpIURcaSzoTVX1rZIMyuP6ndErw3r70TyuFXSbcBe4IaI2AkMAT+qVoiIfZJeAc4GdnUgxsJ5johZ/+jFH5rtTh5fAH4GvAJcATwg6d3AAuBwXd3DwML6HUhaD6wHWL58eaHBFsX3EjGzbtfWPo+I+HFEHImIExFxF/AY8DFgAhioqz4AvK3JKiK2RMRwRAwPDg4WH3QBPEfErLvMdJJfL+t0n0cAAkaBNdWNks4A5gBPdSiuQvXq0D2zXuSWgsbaduUhaZGkSyXNlTRb0gjwAeAhYDvwcUnvlzQf+DKwo8yd5Vn06tA9s17kloLG2tlsdSpwCzAOHAQ+B1wWEXsjYhT4LEkS+RVJX8c1bYytrXp16J5ZL3JLQWNta7aKiHHgPZOU/xXwV+2Kp5N6deieWS8q88q2ndTpPo++1YtD98zKKsvQ+OsuXfWWPg9wSwE4eZhZj8va4e2WgsacPMysp2Vd1RbcUtCIk0eX8gx16zczPefd4V0MJ48u5HHn1m+ynPPu8C6G7+fRhTzu3PpNlnPeQ+OL4SuPLpTHZbibvaybZDnn3eFdDCePLpT1MjyPZi8nH2unrOe8O7zz52arLpT1Mjxrs1c1+Tzz4nGCN5OPF4uzorjpqXx85dGFsl6GZ232ymPoo/WfLFerbnoqHyePLpXlMjxrE4CHPlqr8mgqddNTubjZqg9lbQLIuiqw743QfzxCsPf4yqMPZW0CyLLWjzvru1eW4+6r1d7j5NGnsjQBZEk+WftLPEGyM7Ied0/U6z1OHjYjM00+7qzvTlmPu1em7T1OHtZW7qzvnE42O3m0VO9x8rC2yvoLNI/mj27uM5lp7GVodvJoqd7S36Ottm+HlSth1qzkz+3bOx1Rz7vsvGXcevm5LFs0DwHLFs3j1svPbamzPstIsW6e4Jgl9qyjnTxJz+qV5spD0mnAncCHSe5x/sX01rTF2L4d1q+HY8eS12NjyWuAkZHp7+OGG+DAAVi+HDZvnv57+1inOuur78vaYZ/lqiXL+7PE7mYny1tpkgfwl8ArwBLg3cDfSdodEaOFfNoNN7yZOKqOHUu2TycBZE0+TjwzliX5ZPkSzdr0k/X9WWJ3s5PlrRTNVpLmA2uBTRExERGPAn8DfKqwDz1woLXt9SZLPlOpJp6xMYh4M/G42axwWSY4Zm36yfr+LLG72cnyVorkAZwNvB4RT9Vs2w0M1VeUtF5SRVJlfHx85p+4fHlr2+tlST5ZEo9lkuVLNGvTT9b3Z4k9a1+TWb2yJI8FwOG6bYeBhfUVI2JLRAxHxPDg4ODMP3HzZnjHO9667R3vSLZPR5bkk/WqB9zZP0NZvkSzLsuS9f1ZE8Bl5y3jsev/gKdv+0Meu/4PnDgsk7L0eUwAA3XbBoAjhX1itX9hpv0Omze/tc8Dpp98li9PmqoabZ+OPDr7+9hM2+6zDjPOY6Kc+x2sLMpy5fEUMFvSWTXb1gDFdJZXjYzA/v1w8mTyZytfvCMjsGULrFgBUvLnli3T20fWqx43e3VEHr/83XRkvUIR0ekYAJD010AAnyYZbfX3wHsnG201PDwclUqlTRHmLMtoq1mzko72elKSCIv+fDPrapJ2RcRwln2UpdkK4BpgK/Ar4NfAnxU2TLcMRkZm/mXtZi8z67CyNFsREb+JiMsiYn5ELC90gmC3c7OXmXVYaZKHtSBLfwvkM9rLzPqak0e3ytLZn3WOC3iosFmfc/LoR1mbvTxD3qzvOXn0o6zNXu4zMet7pRmqOxNdPVS3m+UxVNjMOiaPobq+8rDW5dFnYmZdzcnDWpdHn4k72826mpOHtS5Ln4k72816gvs8rL1Wrmw8O37FimTIsZkVzn0e1n08QdGsJzh5WHu5s92sJzh5WHtl7Ww3s1Jw8rD2yjpBETxay6wEyrQku/WLLMvRezl5s1LwlYd1Fy+NYlYKTh7WXTxay6wUnDysu3i0llkptCV5SNop6WVJE+ljb135JySNSToq6X5Jp7UjLutCHq1lVgrtvPLYEBEL0seq6kZJQ8C3gE8BS4BjwNfbGJd1kzxGa5lZZmVothoBHoiIH0TEBLAJuFzSwg7HZWWV5S6K4KG+ZjloZ/K4VdJBSY9Juqhm+xCwu/oiIvYBrwBnN9qJpPWSKpIq4+PjhQZsPcgLM5rlol3J4wvAGcAyYAvwgKQz07IFwOG6+oeBhlceEbElIoYjYnhwcLCoeK1XeaivWS4yJ4+0MzyaPB4FiIgfR8SRiDgREXcBjwEfS3cxAQzU7XYAOJI1NrO38VBfs1xknmEeERfN5G2A0uejwJpqgaQzgDnAU1ljM3ub5csbLwnvob5mLSm82UrSIkmXSporabakEeADwENple3AxyW9X9J84MvAjojwlYflz0N9zXLRjj6PU4FbgHHgIPA54LKI2AsQEaPAZ0mSyK9I+jquaUNc1o881NcsF76ToFkrtm9POtcPHEiaujZvduKxrpPHnQS9qq7ZdHlFX7M3lGGSoFl38DBfszc4eZhNl4f5mr3BycNsuryir9kbnDzMpsvDfM3e4ORhNl0e5mv2Bo+2MmtFlvuvm/UQX3mYmVnLnDzM2sn3ErEe4WYrs3bxJEPrIb7yMGsXTzK0HuLkYdYunmRoPcTJw6xdPMnQeoiTh1m7eJKh9RAnD7N28SRD6yEebWXWTp5kaD3CVx5mZtayXJKHpA2SKpJOSNrWoPxiSXskHZP0sKQVNWVzJG2V9JKk5yVdm0dMZj3JkwytJPK68niW5D7lW+sLJJ0O7AA2AacBFeA7NVVuAs4CVgAfAj4v6SM5xWXWO6qTDMfGIOLNSYZOINYBuSSPiNgREfcDv25QfDkwGhH3RsTLJMlijaTVaflVwM0RcSgingTuANblEZdZT/EkQyuRdvR5DAG7qy8i4iiwDxiStBhYWluePh9qtjNJ69Mmssr4+HhBIZuVkCcZWom0I3ksAA7XbTsMLEzLqCuvljUUEVsiYjgihgcHB3MN1KzUPMnQSmTK5CFpp6Ro8nh0Gp8xAQzUbRsAjqRl1JVXy8yslicZWolMmTwi4qKIUJPHhdP4jFFgTfWFpPnAmST9IIeA52rL0+ejrf01zPqAJxlaieQ1VHe2pLnAKcApkuZKqk5AvA84R9LatM6XgCciYk9afjewUdLitBP9amBbHnGZ9ZyREdi/H06eTP504rAOyavPYyNwHLge+GT6fCNARIwDa4HNwCHgAuCKmvfeSNKBPgY8AtweEQ/mFJeZmRVAEdHpGGZseHg4KpVKp8MwM+sqknZFxHCWfXh5ErN+4dnpliMvjGjWD3wLXMuZrzzM+oFnp1vOnDzM+oFnp1vOnDzM+oFnp1vOnDzM+oFnp1vOnDzM+oFnp1vOPNrKrF/4FriWI195mJlZy5w8zMysZU4eZmbWMicPM5seL29iNdxhbmZT8/ImVsdXHmY2NS9vYnWcPMxsal7exOo4eZjZ1Ly8idVx8jCzqXl5E6uT1z3MN0iqSDohaVtd2UpJIWmi5rGppnyOpK2SXpL0vKRr84jJzHLk5U2sTl6jrZ4FbgEuBeY1qbMoIl5rsP0m4CxgBfAu4GFJP/N9zM1KxsubWI1crjwiYkdE3A/8egZvvwq4OSIORcSTwB3AujziMjOzYrSzz2NM0i8lfVvS6QCSFgNLgd019XYDQ812Iml92kRWGR8fLzZiMzNrqB3J4yDwHpJmqfOBhUB1auqC9M/DNfUPp3UaiogtETEcEcODg4MFhGtmZlOZMnlI2pl2eDd6PDrV+yNiIiIqEfFaRLwAbAA+LGkAmEirDdS8ZQA4MpO/jJmVmJc36SlTdphHxEU5f2akfyoiDkl6DlgD/GO6fQ0wmvNnmlkneXmTnpPXUN3ZkuYCpwCnSJoraXZadoGkVZJmSXon8DVgZ0RUm6ruBjZKWixpNXA1sC2PuMysJLy8Sc/Jq89jI3AcuB74ZPp8Y1p2BvAgSVPUT4ETwJU1770R2AeMAY8At3uYrlmP8fImPUcRMXWtkhoeHo5KpdLpMMxsKitXJk1V9VasgP372x1N35O0KyKGs+zDy5OYWfG8vEnPcfIws+J5eZOe45tBmVl7eHmTnuIrDzMza5mTh5mZtczJw8zMWubkYWZmLXPyMDOzljl5mFl38MKKpeKhumZWfl5YsXR85WFm5eeFFUvHycPMys8LK5aOk4eZld/y5a1tt8I5eZhZ+XlhxdJx8jCz8vPCiqXj0VZm1h28sGKp+MrDzMxaljl5SJoj6U5JY5KOSHpc0kfr6lwsaY+kY5IelrSi7v1bJb0k6XlJ12aNyczMipXHlcds4BfAB4HfAjYB35W0EkDS6cCOdPtpQAX4Ts37bwLOAlYAHwI+L+kjOcRlZmYFyZw8IuJoRNwUEfsj4mRE/C3wNHB+WuVyYDQi7o2Il0mSxRpJq9Pyq4CbI+JQRDwJ3AGsyxqXmZkVJ/c+D0lLgLOB0XTTELC7Wh4RR4F9wJCkxcDS2vL0+dAk+18vqSKpMj4+nnf4ZmY2DbkmD0mnAtuBuyJiT7p5AXC4ruphYGFaRl15tayhiNgSEcMRMTw4OJhP4GbW27yoYu6mTB6SdkqKJo9Ha+rNAu4BXgE21OxiAhio2+0AcCQto668WmZmll11UcWxMYh4c1FFJ5BMpkweEXFRRKjJ40IASQLuBJYAayPi1ZpdjAJrqi8kzQfOJOkHOQQ8V1uePh/FzCwPXlSxEHk1W30D+F3g4xFxvK7sPuAcSWslzQW+BDxR06x1N7BR0uK0E/1qYFtOcZlZv/OiioXIY57HCuAzwLuB5yVNpI8RgIgYB9YCm4FDwAXAFTW7uJGkA30MeAS4PSIezBqXmRngRRULknl5kogYAzRFnX8CVjcpOwH8afowM8vX5s1vvZEUeFHFHHh5EjPrbV5UsRBeGNHMep8XVcydrzzMzKxlTh5mZtYyJw8zM2uZk4eZmbXMycPMzFrm5GFmZi1z8jAzm4pX5X0bz/MwM5tMdVXe6gz16qq80NdzR3zlYWY2Ga/K25CTh5nZZLwqb0NOHmZmk/GqvA05eZiZTWbz5mQV3lpeldfJw8xsUl6VtyGPtjIzm4pX5X0bX3mYmVnL8rgN7RxJd0oak3RE0uOSPlpTvlJS1NyedkLSprr3b5X0kqTnJV2bNSYzMytWHs1Ws4FfAB8EDgAfA74r6dyI2F9Tb1FEvNbg/TcBZwErgHcBD0v6me9jbmZWXpmvPCLiaETcFBH7I+JkRPwt8DRw/jR3cRVwc0QciogngTuAdVnjMjOz4uTe5yFpCXA2MFpXNCbpl5K+Len0tO5iYCmwu6bebmAo77jMzCw/uSYPSacC24G7ImJPuvkg8B6SZqnzgYVpHYAF6Z+Ha3ZzOK3T7DPWS6pIqoyPj+cZvpmZTdOUyUPSzrTDu9Hj0Zp6s4B7gFeADdXtETEREZWIeC0iXkjLPixpAJhIqw3UfOQAcKRZPBGxJSKGI2J4cHCwpb+smZnlY8rkEREXRYSaPC4EkCTgTmAJsDYiXp1sl+mfiohDwHPAmpryNby9ycvMrHv14JLueU0S/Abwu8AlEXG8tkDSBcCLwP8FFgNfA3ZGRLWp6m5go6QKSfK5GviTnOIyM+usHl3SPY95HiuAzwDvBp6vmctRPSpnAA+SNEX9FDgBXFmzixuBfcAY8Ahwu4fpmlnP6NEl3RURU9cqqeHh4ahUKp0Ow8ysuVmzoNH3rAQnT7Y/HkDSrogYzrIPL09iZlakHl3S3cnDzKxIPbqku5OHmVmRenRJdy/JbmZWtB5c0t1XHmZm1jInDzMza5mTh5mZtczJw8zMWubkYWZmLXPyMDOzljl5mJlZy5w8zMysZU4eZmZlVtJ7gXiGuZlZWZX4XiC+8jAzK6sS3wvEycPMrKwOHGhtexs5eZiZlVWJ7wXi5GFmVlYlvhdILslD0v+Q9JyklyQ9JenTdeUXS9oj6Zikh9P7nlfL5kjamr73eUnX5hGTmVnXK/G9QHK5h7mkIeD/RcQJSauBncAfRsQuSacD+4BPAw8ANwPvj4jfT997K3Ah8B+BdwEPA+si4sGpPtf3MDcza11p7mEeEaMRcaL6Mn2cmb6+HBiNiHsj4mXgJmBNmmQArgJujohDEfEkcAewLo+4zMysGLnN85D0dZIv/XnA48Dfp0VDwO5qvYg4KmkfMCTpBWBpbXn6/LJJPmc9kA50ZkLS3hzCPx04mMN+ilDm2KDc8Tm2mSlzbFDu+LolthWTVZyO3JJHRFwj6XPAvwMuAqpXIguA8brqh4GFaVn1dX1Zs8/ZAmzJIeQ3SKpkvYQrSpljg3LH59hmpsyxQbnj66fYpmy2krRTUjR5PFpbNyJej4hHgd8B/izdPAEM1O12ADiSllFXXi0zM7OSmjJ5RMRFEaEmjwubvG02b/Z5jAJrqgWS5qdloxFxCHiutjx9PjqTv4yZmbVH5g5zSb8t6QpJCySdIulS4Erg+2mV+4BzJK2VNBf4EvBEROxJy+8GNkpanHaiXw1syxpXi3JtBstZmWODcsfn2GamzLFBuePrm9gyD9WVNAh8j+SKYRYwBnwtIu6oqXMJ8N9JOml+TDIUd39aNgf4BvCfgOPAVyLiv2UKyszMCpXLPA8zM+svXp7EzMxa5uRhZmYt64vkIek0SfdJOippTNInmtSTpK9I+nX6+KokFRzbHEl3pnEdkfS4pI82qbtO0uuSJmoeFxUc305JL9d8XsNJme0+dnXHYCI9Ln/RpG7hx03SBkkVSSckbasra7q2W4P9rEzrHEvfc0lRsUn6fUn/KOk3ksYl3SvpX02yn2mdCznGtzKdElD777Zpkv2089iN1MV1LI31/Cb7yf3YTfXdUfR51xfJA/hL4BVgCTACfEPJelz11pPMbl8D/B7wH4DPFBzbbOAXwAeB3wI2Ad+VtLJJ/f8dEQtqHjsLjg9gQ83nrWpSp63HrvYYkPy7HgfuneQtRR+3Z4FbgK21G5Ws7baD5N/1NKACfGeS/fxPkhUa3gncAHxPyaCU3GMDFpOMwFlJMpjlCPDtKfY1nXMhr/iqFtV85s2T7Kdtxy4ittedg9cAPwf+ZZJ95X3smn53tOW8i4iefgDzSRLH2TXb7gFua1D3R8D6mtf/GfjnDsT8BLC2wfZ1wKNtjmUn8Olp1OvYsQP+mOQ/rpqUt+24kXzRbKt5vR74Uc3r+SSJbnWD955NsjLDwpptPwQ+W0RsDcr/LXAk67mQ47FbSbJO3uxpvLfTx+5h4MZOHbuaz3kCWNuO864frjzOBl6PiKdqtu0mWXOr3lvW4ZqkXmEkLSGJudlEyfMkHVSy9P0mSe24D/2t6Wc+NklzTyeP3R8Dd0d61jfRieMGDdZ2I1llutn59/OIqF1hoZ3H8QNMPUF3OudC3sYk/VLSt9Nf1I107NilzUEfIJmzNplCj13dd0fh510/JI8FvHXtLGi+flZ93cPAgiLb7mtJOhXYDtwVb06irPUD4Bzgt0l+XVwJXFdwWF8AzgCWkTRxPCDpzAb1OnLsJC0nuWy/a5JqnThuVVnOv8nq5krS75FM4J3suEz3XMjLQeA9JE1q55Mch+1N6nbs2JGsDP7DiHh6kjqFHrsG3x2Fn3f9kDwmW1trqroDwMQUv2hzIWkWSXPaK8CGRnUi4ucR8XREnIyInwBfJplcWZiI+HFEHImIExFxF/AY8LEGVTt17K4iaZJq+h+3E8etRpbzb7K6uZH0b4B/AP5LRPywWb0WzoVcRMRERFQi4rWIeIHk/8WHJdUfI+jQsUtdxeQ/Xgo9dk2+Owo/7/oheTwFzJZ0Vs22ZutnvWUdrknq5Sr9dX4nScfv2oh4dZpvDaAtV0XT+MyOHDum8R+3gXYet6ZruzWpe4ak2l98hR7HtMnln0juqXNPi29v9/lX/SHS7Pxr67EDkPQ+kttKfK/Ft+Zy7Cb57ij+vCu6A6cMD+CvSUYTzAfeR3JJNtSg3meBJ0kuLZemBy+XDrcp4vsm8M/AginqfRRYkj5fDfyUSTrpcohrEXApMJdkZMcIcBRYVYZjB7w3jWfhFPUKP27p8ZkL3EryK7B6zAbT821tuu0rTDKQID0P/jyt+0fAi8BgQbEtI2kHvy7PcyHH+C4AVpH8yH0nyWihh8tw7GrKt5D0t3Xq2DX87mjHeZfbf54yP0iGqt2f/oMdAD6Rbn8/SdNKtZ6ArwK/SR9fpckInhxjW0HyK+RlksvH6mMEWJ4+X57W/XPghfTv8XOS5pdTC4xtEPg/JJevL6Yn2L8v0bH7FnBPg+1tP24kd8iMusdNadklwB6S0S47gZU17/sm8M2a1yvTOseBvcAlRcUG3Jg+rz3vav9N/yvwD1OdCwXGdyXwdPrv9hxJh/S7ynDs0rK56bG4uMH7Cj92TPLd0Y7zzmtbmZlZy/qhz8PMzHLm5GFmZi1z8jAzs5Y5eZiZWcucPMzMrGVOHmZm1jInDzMza5mTh5mZtez/A8bIaIXveo0oAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_preds(preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 3: Calculate the loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(25823.8086, grad_fn=<MeanBackward0>)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss = mse(preds, speed)\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 4: Calculate the gradients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-53195.8594,  -3419.7146,   -253.8908])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss.backward()\n",
    "params.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-0.5320, -0.0342, -0.0025])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params.grad * 1e-5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-0.7658, -0.7506,  1.3525], requires_grad=True)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 5: Step the weights. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 1e-5\n",
    "params.data -= lr * params.grad.data\n",
    "params.grad = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(5435.5366, grad_fn=<MeanBackward0>)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds = f(time,params)\n",
    "mse(preds, speed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAEACAYAAABLfPrqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAcYElEQVR4nO3df4wc5Z3n8ffHGGFje9YmzJKzVx4vHNgrwzqcJ+IO8sMJuZCwx4nF9wdkEta3G5wscnQSJxJy2AEFkElY3R/Zu/wwgpgfXjYhMmhJduGyG0yCcxvdEGSSCTaSA0P4mXFijMc25tf3/qjq0HS6Z6amqrpruj8vqTTd9TxV/Z1navrbz/NUVSsiMDMzy2JWpwMwM7OZx8nDzMwyc/IwM7PMnDzMzCwzJw8zM8vMycPMzDJz8jAzs8wKTR6SNkgalnRU0taGsnMl7ZZ0WNKDkgbqyo6TdKuklyW9IOmKIuMyM7NiFd3zeA64Hri1fqWkE4HtwCbgBGAY+FZdlWuBU4EB4APAZyV9pODYzMysICrjCnNJ1wN/FBHr0ufrgXURcXb6fB6wDzgzInZLehb4rxHxf9Ly64BTI+LiwoMzM7PcZrfpdVYCu2pPIuKQpL3ASkkvAovry9PHFzbbUZqI1gPMmzdv9YoVK0oL2sysGz3yyCP7IqI/zz7alTzmA2MN6w4AC9Ky2vPGst8TEVuALQCDg4MxPDxcbKRmZl1O0mjefbTrbKtxoK9hXR9wMC2jobxWZmZmFdSu5DECrKo9Sec8TgFGImI/8Hx9efp4pE2xmZlZRkWfqjtb0hzgGOAYSXMkzQbuAU6XtDYt/wLwWETsTje9HdgoaZGkFcBlwNYiYzMzs+IU3fPYCBwBrgI+nj7eGBFjwFrgBmA/cBZQfybVNcBeYBR4CLgpIu4vODYzMytIKafqtosnzM3MspP0SEQM5tmHb09iZmaZOXmYmVlmTh5mZpaZk4eZmWXWrivMK+neR5/lpgf28NxLR1i8cC5XnrecC89c0umwzMwqr2eTx72PPsvnt/+MI6+9AcCzLx3h89t/BuAEYmY2iZ4dtrrpgT2/Sxw1R157g5se2NOhiMzMZo6eTR7PvXQk03ozM3tLzw5bLV44l2ebJIrFC+d2IBozs+aqOjfbsz2PK89bztxjj3nburnHHsOV5y3vUERmZm9Xm5t99qUjBG/Nzd776LOdDq13k8eFZy5h80VnsGThXAQsWTiXzRedUYmMbmYG1Z6b7dlhK0gSiJOFmVVVledmezp55FXVsUgz6w5Vnpvt2WGrvKo8Fmlm1XLvo89yzo0/4I+v+h7n3PiDKb9PVHlu1sljmqo8Fmlm1ZHng2aV52Y9bDVNVR6LNLPqmOiD5lSSQFXnZt3zmKZWY45VGIs0s+ro1g+abU0eknZIekXSeLrsqSv7mKRRSYck3SvphHbGllWVxyLNrDq69YNmJ3oeGyJifrosB5C0EvgG8AngJOAw8NUOxDZlVR6LNLPq6NYPmlWZ8xgC7ouIHwJI2gQ8LmlBRBzsbGitVXUs0syqo/Ye0W2n9XcieWyWdCOwB7g6InYAK4Ef1ypExF5JrwKnAY90IMbS+RoRs97RjR802508Pgf8AngVuBi4T9K7gPnAgYa6B4AFjTuQtB5YD7B06dJSgy2Lv0vEzGa6ts55RMRPIuJgRByNiNuAncD5wDjQ11C9D/i9IauI2BIRgxEx2N/fX37QJfA1ImYzy3Qv8utmnZ7zCEDACLCqtlLSycBxwBMdiqtU3Xrqnlk38khBc23reUhaKOk8SXMkzZY0BLwPeADYBlwg6b2S5gFfBLZXebI8j249dc+sG3mkoLl2DlsdC1wPjAH7gM8AF0bEnogYAT5NkkR+TTLXcXkbY2urbj11z6wbeaSgubYNW0XEGPDuCcr/Dvi7dsXTSd166p5ZN6rynW07qdNzHj2rG0/dM6uqPKfGX3ne8rfNeYBHCsDJw8y6XN4Jb48UNOfkYWZdLe9dbcEjBc04ecxQvkLdes10j3lPeJfDyWMG8nnn1mvyHPOe8C6Hv89jBvJ559Zr8hzzPjW+HO55zEBFdMM97GUzSZ5j3hPe5XDymIHydsOLGPZy8rF2ynvMe8K7eB62moHydsPzDnvVks+zLx0heCv5+GZxVhYPPVWPex4zUN5ueN5hryJOfbTek6e36qGn6nHymKHydMPzDgH41EfLqoihUg89VYuHrXpQ3iGAvHcF9ncj9B6fIdh93PPoQXmHAPLc68eT9TNXnnZ3b7X7OHn0qDxDAHmST975El8g2Rl5290X6nUfJw+blukmH0/Wz0x52913pu0+Th7WVp6s75xODjv5bKnu4+RhbZX3E2gRwx8zec5kurFXYdjJZ0t1l94+22rbNli2DGbNSn5u29be7XvQhWcuYfNFZ7Bk4VwELFk4l80XnZFpsj7PmWIz+QLHPLHnPdvJF+lZo8r0PCSdANwCfJjkO84/n341bTm2bYP16+Hw4eT56GjyHGBoqD3bX301PP00LF0KN9wwte26QKcm62vb5Z2wz9NrybN9ntg97GRFU0R0OgYAJN1F0hP6K+BdwPeAsyNipNU2g4ODMTw8PL0XXLYsecNvNDAATz1V7vaNiQfg+ONhy5apJ5AeTj55/PFV36PZES/gyRv/bMJtG4d+IPn0PdWeU97t88R+zo0/aDrstGThXHZe9cFJX9u6i6RHImIwzz4qMWwlaR6wFtgUEeMR8TDwD8AnSnvRp5/Otr7I7a+++u2JA5LnV189tdeuJZ/RUYh4q9fjYbNJ5bnAMe/QT97t88TuYScrWiWSB3Aa8EZEPFG3bhewsrGipPWShiUNj42NTf8Vly7Ntr7I7fMmrrzJB3p2vibPm2jeoZ+82+eJPe9ck1mjqiSP+cCBhnUHgAWNFSNiS0QMRsRgf3//9F/xhhuSoaJ6xx+frC97+7yJK2/yKaLnMkOTT5430by3Zcm7fd4EcOGZS9h51Qd58sY/Y+dVH3TisFwqMech6UxgZ0QcX7fuvwNrIuKCVtvlmvOA/PMG090+75xHJ+droJg5mxmo03MeZkUpYs6DiOj4AswDXgVOrVt3O3DjRNutXr06Zqw774wYGIiQkp933plt2+OPj0j6Dcly/PFT34f09m1rizS17QcGmm8/MDD1+Kf7u3fYPT99Js7e/C+x7HPfjbM3/0vc89Nn2rq9WRGA4cj5vl2JngeApL8HAvgkydlW/0iZZ1vNdHl6TXl7HrNmJemikQRvvjnxtj3aazGrkq452yp1OTAX+DVwF/DXEyWOnjc0lLzRv/lm8jPLG2/e+Z48czae7DfrCpVJHhHx24i4MCLmRcTSKPMCwV43NJR80h8YSHoLAwPZPvnnST5VmOw3s9wqkzyszfL0XPIkn7xnmrnnYlYJTh42PdNNPnmHzNxzMasEJw9rr7xDZlXouZiZk4d1QCcn+4vouXjIy8zJw2aYTvZcPORl9juVuc5jOnr6Og+bnjzXmeS9PsasIrrtOg+z8uXpueQd8gIPe1nXqMyXQZm1zdDQ9K5mX7q0ec9jqpP1eb9AzKxC3PMwm6q8k/U+08u6iJOH2VTlnaz3sJd1EQ9bmWUx3SEv8LCXdRX3PMzaxcNe1kWcPMzapdPDXh7ysgJ52MqsnTo17OUhLyuYex5mM0WeYS8PeVnBnDzMZopOX+BoVsfJw2wmme5NJfPejRg8Z2Jv05bkIWmHpFckjafLnobyj0kalXRI0r2STmhHXGY9I++ZXr4ppDVoZ89jQ0TMT5fltZWSVgLfAD4BnAQcBr7axrjMul/eM708Z2INqjBsNQTcFxE/jIhxYBNwkaQFHY7LrLvk+R4VXx1vDdqZPDZL2idpp6Q1detXArtqTyJiL/AqcFqznUhaL2lY0vDY2FipAZtZKu+ciYe9uk67ksfngJOBJcAW4D5Jp6Rl84EDDfUPAE17HhGxJSIGI2Kwv7+/rHjNrJ6vjrcGuZNHOhkeLZaHASLiJxFxMCKORsRtwE7g/HQX40Bfw277gIN5YzOzgnT66nirnNzJIyLWRIRaLO9ptRmg9PEIsKpWIOlk4DjgibyxmVmB8syZ+FThrlP6sJWkhZLOkzRH0mxJQ8D7gAfSKtuACyS9V9I84IvA9ohwz8OsW/hU4a7TjjmPY4HrgTFgH/AZ4MKI2AMQESPAp0mSyK9J5joub0NcZtYuPlW46ygiOh3DtA0ODsbw8HCnwzCzss2alfQ4GknJMJplIumRiBjMs48qXOdhZjaxIuZMrFBOHmZWfUXMmXiyvVBOHmZWfXnmTDzZXgrPeZhZd1u2rPmXaA0MJKcc9yDPeZiZTcYXKJbCycPMupsvUCyFk4eZdTdfoFgKJw8z626+QLEUnjA3M5tIF16g6AlzM7Oy+QLFppw8zMwmknfOBLpywt3Jw8xsInnnTLp0wt1zHmZmZargRYqe8zAzq7ouvUjRycPMrExdOuHu5GFmVqYuvSOwk4eZWZm69I7AhSQPSRskDUs6Kmlrk/JzJe2WdFjSg5IG6sqOk3SrpJclvSDpiiJiMjOrjKGhZHL8zTeTn11wdXtRPY/nSL6n/NbGAkknAtuBTcAJwDDwrboq1wKnAgPAB4DPSvpIQXGZmc1cFZ5sLyR5RMT2iLgX+E2T4ouAkYi4OyJeIUkWqyStSMsvBa6LiP0R8ThwM7CuiLjMzGa0Ck+2t2POYyWwq/YkIg4Be4GVkhYBi+vL08crW+1M0vp0iGx4bGyspJDNzCqgiKvbS9KO5DEfONCw7gCwIC2jobxW1lREbImIwYgY7O/vLzRQM7NKyXt1e4lmT1ZB0g7g/S2Kd0bEeybZxTjQ17CuDziYltWev9JQZmZmQ0OVSBaNJu15RMSaiFCLZbLEATACrKo9kTQPOIVkHmQ/8Hx9efp4JNuvYWZm7VTUqbqzJc0BjgGOkTRHUq1Xcw9wuqS1aZ0vAI9FxO60/HZgo6RF6ST6ZcDWIuIyM7NyFDXnsRE4AlwFfDx9vBEgIsaAtcANwH7gLODium2vIZlAHwUeAm6KiPsLisvMzErgu+qamfUY31XXzMw6wsnDzMwyc/IwM7PMnDzMzCwzJw8zM8vMycPMzDJz8jAzs8ycPMzMLDMnDzMzy8zJw8zMMnPyMDOzzJw8zMwsMycPMzPLzMnDzMwyc/IwM7PMnDzMzCwzJw8zM8usqO8w3yBpWNJRSVsbypZJCknjdcumuvLjJN0q6WVJL0i6ooiYzMysPLML2s9zwPXAecDcFnUWRsTrTdZfC5wKDADvBB6U9At/j7mZWXUV0vOIiO0RcS/wm2lsfilwXUTsj4jHgZuBdUXEZWZm5WjnnMeopGckfVPSiQCSFgGLgV119XYBK1vtRNL6dIhseGxsrNyIzcysqXYkj33Au0mGpVYDC4Btadn89OeBuvoH0jpNRcSWiBiMiMH+/v4SwjUzs8lMmjwk7UgnvJstD0+2fUSMR8RwRLweES8CG4APS+oDxtNqfXWb9AEHp/PLmJlZe0w6YR4Rawp+zUh/KiL2S3oeWAV8P12/Chgp+DXNzKxARZ2qO1vSHOAY4BhJcyTNTsvOkrRc0ixJ7wC+AuyIiNpQ1e3ARkmLJK0ALgO2FhGXmZmVo6g5j43AEeAq4OPp441p2cnA/SRDUT8HjgKX1G17DbAXGAUeAm7yabpmZtWmiJi8VkUNDg7G8PBwp8MwM5tRJD0SEYN59uHbk5iZWWZOHmZmlpmTh5mZZebkYWZmmTl5mJlZZk4eZmaWmZOHmZll5uRhZmaZOXmYmVlmTh5mZpaZk4eZmWXm5GFmZpk5eZiZWWZOHmZmlpmTh5mZZebkYWZmmTl5mJlZZrmTh6TjJN0iaVTSQUmPSvpoQ51zJe2WdFjSg5IGGra/VdLLkl6QdEXemMzMrFxF9DxmA78C3g/8AbAJ+LakZQCSTgS2p+tPAIaBb9Vtfy1wKjAAfAD4rKSPFBCXmZmVJHfyiIhDEXFtRDwVEW9GxHeBJ4HVaZWLgJGIuDsiXiFJFqskrUjLLwWui4j9EfE4cDOwLm9cZmZWnsLnPCSdBJwGjKSrVgK7auURcQjYC6yUtAhYXF+ePl45wf7XSxqWNDw2NlZ0+GZmNgWFJg9JxwLbgNsiYne6ej5woKHqAWBBWkZDea2sqYjYEhGDETHY399fTOBmZpbJpMlD0g5J0WJ5uK7eLOAO4FVgQ90uxoG+ht32AQfTMhrKa2VmZlZRkyaPiFgTEWqxvAdAkoBbgJOAtRHxWt0uRoBVtSeS5gGnkMyD7Aeery9PH49gZmaVVdSw1deAPwEuiIgjDWX3AKdLWitpDvAF4LG6Ya3bgY2SFqWT6JcBWwuKy8zMSlDEdR4DwKeAdwEvSBpPlyGAiBgD1gI3APuBs4CL63ZxDckE+ijwEHBTRNyfNy4zMyvP7Lw7iIhRQJPU+WdgRYuyo8BfpouZmc0Avj2JmZll5uRhZmaZOXmYmVlmTh5mZpaZk4eZmWXm5GFmZpk5eZiZWWZOHmZmlpmTh5mZZebkYWZmmTl5mJlZZk4eZmaWmZOHmZll5uRhZmaZOXmYmVlmTh5mZpaZk4eZmWVWxNfQHifpFkmjkg5KelTSR+vKl0mKuq+nHZe0qWH7WyW9LOkFSVfkjcnMzMqV+2to0338Cng/8DRwPvBtSWdExFN19RZGxOtNtr8WOBUYAN4JPCjpF/4eczOz6srd84iIQxFxbUQ8FRFvRsR3gSeB1VPcxaXAdRGxPyIeB24G1uWNy8zMylP4nIekk4DTgJGGolFJz0j6pqQT07qLgMXArrp6u4CVRcdlZmbFKTR5SDoW2AbcFhG709X7gHeTDEutBhakdQDmpz8P1O3mQFqn1WuslzQsaXhsbKzI8M3MbIomTR6SdqQT3s2Wh+vqzQLuAF4FNtTWR8R4RAxHxOsR8WJa9mFJfcB4Wq2v7iX7gIOt4omILRExGBGD/f39mX5ZMzMrxqQT5hGxZrI6kgTcApwEnB8Rr020y9pmEbFf0vPAKuD76fpV/P6Ql5mZVUhRw1ZfA/4EuCAijtQXSDpL0nJJsyS9A/gKsCMiakNVtwMbJS2StAK4DNhaUFxmZlaCIq7zGAA+BbwLeKHuWo6htMrJwP0kQ1E/B44Cl9Tt4hpgLzAKPATc5NN0zcyqLfd1HhExCmiC8ruAuyYoPwr8ZbqYmdkM4NuTmJlZZk4eZmaWmZOHmZll5uRhZmaZOXmYmVlmTh5mZpaZk4eZmWXm5GFmZpk5eZiZWWZOHmZmlpmTh5mZZebkYWZmmTl5mJlZZk4eZmaWmZOHmZll5uRhZmaZOXmYmVlmTh5mZpZZIclD0p2Snpf0sqQnJH2yofxcSbslHZb0YPq957Wy4yTdmm77gqQriojJzMzKU1TPYzOwLCL6gP8MXC9pNYCkE4HtwCbgBGAY+FbdttcCpwIDwAeAz0r6SEFxmZlZCQpJHhExEhFHa0/T5ZT0+UXASETcHRGvkCSLVZJWpOWXAtdFxP6IeBy4GVhXRFxmZlaO2UXtSNJXSd705wKPAv+YFq0EdtXqRcQhSXuBlZJeBBbXl6ePL5zgddYD69On45L2FBD+icC+AvZThirHBtWOz7FNT5Vjg2rHN1NiG5io4lQUljwi4nJJnwH+A7AGqPVE5gNjDdUPAAvSstrzxrJWr7MF2FJAyL8jaTgiBovcZ1GqHBtUOz7HNj1Vjg2qHV8vxTbpsJWkHZKixfJwfd2IeCMiHgb+CPjrdPU40New2z7gYFpGQ3mtzMzMKmrS5BERayJCLZb3tNhsNm/NeYwAq2oFkualZSMRsR94vr48fTwynV/GzMzaI/eEuaQ/lHSxpPmSjpF0HnAJ8IO0yj3A6ZLWSpoDfAF4LCJ2p+W3AxslLUon0S8DtuaNK6NCh8EKVuXYoNrxObbpqXJsUO34eiY2RUS+HUj9wHdIegyzgFHgKxFxc12dDwH/i2SS5ifAuoh4Ki07Dvga8F+AI8CXIuJ/5grKzMxKlTt5mJlZ7/HtSczMLDMnDzMzy6wnkoekEyTdI+mQpFFJH2tRT5K+JOk36fJlSSo5tuMk3ZLGdVDSo5I+2qLuOklvSBqvW9aUHN8OSa/UvV7TizLb3XYNbTCetsvftqhbertJ2iBpWNJRSVsbylre263JfpaldQ6n23yorNgk/XtJ35f0W0ljku6W9G8m2M+UjoUC41uWXhJQ/3fbNMF+2tl2Qw1xHU5jXd1iP4W33WTvHWUfdz2RPID/DbwKnAQMAV+TtLJJvfUkV7evAv4U+E/Ap0qObTbwK+D9wB+Q3APs25KWtaj/fyNift2yo+T4ADbUvd7yFnXa2nb1bUDydz0C3D3BJmW323PA9cCt9Ss1+b3dGt1FcoeGdwBXA99RclJK4bEBi0jOwFlGcjLLQeCbk+xrKsdCUfHVLKx7zesm2E/b2i4itjUcg5cDvwR+OsG+im67lu8dbTnuIqKrF2AeSeI4rW7dHcCNTer+GFhf9/yvgH/tQMyPAWubrF8HPNzmWHYAn5xCvY61HfAXJP+4alHetnYjeaPZWvd8PfDjuufzSBLdiibbnkZyZ4YFdet+BHy6jNialP874GDeY6HAtltGcp+82VPYttNt9yBwTafaru51HgPWtuO464Wex2nAGxHxRN26XST33Gr0tvtwTVCvNJJOIom51YWSZ0rap+TW95skFXaLmQlsTl9z5wTDPZ1su78Abo/0qG+hE+0GTe7tBuyl9fH3y4iov8NCO9vxfUx+ge5UjoWijUp6RtI300/UzXSs7dLhoPeRXLM2kVLbruG9o/TjrheSx3zefu8saH3/rMa6B4D5ZY7d15N0LLANuC3euoiy3g+B04E/JPl0cQlwZclhfQ44GVhCMsRxn6RTmtTrSNtJWkrSbb9tgmqdaLeaPMffRHULJelPSS7gnahdpnosFGUf8G6SIbXVJO2wrUXdjrUdyZ3BfxQRT05Qp9S2a/LeUfpx1wvJY6J7a01Wtw8Yn+QTbSEkzSIZTnsV2NCsTkT8MiKejIg3I+JnwBdJLq4sTUT8JCIORsTRiLgN2Amc36Rqp9ruUpIhqZb/uJ1otzp5jr+J6hZG0r8F/gn4bxHxo1b1MhwLhYiI8YgYjojXI+JFkv+LD0tqbCPoUNulLmXiDy+ltl2L947Sj7teSB5PALMlnVq3rtX9s952H64J6hUq/XR+C8nE79qIeG2KmwbQll7RFF6zI23HFP5xm2hnu7W8t1uLuidLqv/EV2o7pkMu/0zynTp3ZNy83cdf7YNIq+OvrW0HIOkckq+V+E7GTQtpuwneO8o/7sqewKnCAvw9ydkE84BzSLpkK5vU+zTwOEnXcnHaeIVMuE0S39eBfwXmT1Lvo8BJ6eMVwM+ZYJKugLgWAucBc0jO7BgCDgHLq9B2wNlpPAsmqVd6u6XtM4fkWzXvqGuz/vR4W5uu+xITnEiQHgd/k9b9c+AloL+k2JaQjINfWeSxUGB8ZwHLST7kvoPkbKEHq9B2deVbSObbOtV2Td872nHcFfbPU+WF5FS1e9M/2NPAx9L17yUZWqnVE/Bl4Lfp8mVanMFTYGwDJJ9CXiHpPtaWIWBp+nhpWvdvgBfT3+OXJMMvx5YYWz/w/0i6ry+lB9h/rFDbfQO4o8n6trcbyTdkRsNybVr2IWA3ydkuO0i+srm23deBr9c9X5bWOQLsAT5UVmzANenj+uOu/m/6P4B/muxYKDG+S4An07/b8yQT0u+sQtulZXPStji3yXaltx0TvHe047jzva3MzCyzXpjzMDOzgjl5mJlZZk4eZmaWmZOHmZll5uRhZmaZOXmYmVlmTh5mZpaZk4eZmWX2/wExahUKOb/yYgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_preds(preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def apply_step(params, prn=True):\n",
    "    preds = f(time, params)\n",
    "    loss = mse(preds, speed)\n",
    "    loss.backward()\n",
    "    params.data -= lr * params.grad.data\n",
    "    params.grad = None\n",
    "    if prn: print(loss.item())\n",
    "    return preds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 6: Repeat the process "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5435.53662109375\n",
      "1577.4495849609375\n",
      "847.3780517578125\n",
      "709.22265625\n",
      "683.0757446289062\n",
      "678.12451171875\n",
      "677.1839599609375\n",
      "677.0025024414062\n",
      "676.96435546875\n",
      "676.9537353515625\n"
     ]
    }
   ],
   "source": [
    "for i in range(10): apply_step(params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "params = orig_params.detach().requires_grad_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1QAAADMCAYAAAB0vOLuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3df4zc9Z3f8dfbLIe9BsemGFJMdn1CHD4Z5CA2ihroBUJVuOS4InxSaCbiSARbtUcvVw6DqY0hgIuJo2suIuW0BMQB2wuXE6Y6LgdVjx8VrlplCYHUxERK8RpsI0xrG/Can/vuH98ZdnY83+/MfOczM98fz4c0Ws/3M7P+7tf78nfe38/n+/mYuwsAAAAA0LkFg94BAAAAAMgrCioAAAAASImCCgAAAABSoqACAAAAgJQoqAAAAAAgJQoqAAAAAEiJggoAAAAAUgpaUJnZtWY2ZWbvm9kDDW0XmdlOM5sxs6fNbLSu7Tgzu9/M3jazN8zsupD7BeQReQLCIlNAWGQKiITuodor6Q5J99dvNLOTJD0q6WZJJ0qakvRI3UtulXSGpFFJF0q6wcwuCbxvQN6QJyAsMgWERaYASebu4b+p2R2STnP3q6rPxyVd5e5fqD5fLOktSee4+04z2yPpG+7+X6vtt0s6w92vCL5zQM6QJyAsMgWERaZQdv26h2q1pBdrT9z9sKRfS1ptZssknVrfXv3z6j7tG5A35AkIi0wBYZEplMpQn/6e4yXtb9h2SNIJ1bba88a2o1SveoxL0uLFi89dtWpV2D0Fmnj++effcvflg96PqmB5ksgUBqOomSJPGBQyBYTVSab6VVC9K2lJw7Ylkt6pttWev9fQdhR3n5A0IUljY2M+NTUVfGeBRmY2Peh9qBMsTxKZwmAUNVPkCYNCpoCwOslUv4b87ZC0pvakOpb2dEk73P2ApH317dU/7+jTvgF5Q56AsMgUEBaZQqmEnjZ9yMwWSjpG0jFmttDMhiRtk3SWma2ttm+S9JK776y+9UFJG81smZmtknSNpAdC7huQN+QJCItMAWGRKSASuodqo6QjktZL+nr1zxvdfb+ktZI2Szog6fOS6mdyuUXRzYrTkp6VtNXdnwi8b0DekCcgLDIFhEWmAPVo2vR+YSwt+sXMnnf3sUHvR6+RKfRLGTJFntBPZAoIq5NM9eseKgAAAAAonH7N8jcQj72wR1uffEV7Dx7RqUsXad3FZ+qyc1YMereA3CJTQDjkCQiLTGFQCltQPfbCHt306C905MOPJUl7Dh7RTY/+QpIIF5ACmQLCIU9AWGQKg1TYIX9bn3zlk1DVHPnwY2198pUB7RGQb2QKCIc8AWGRKQxSYXuo9h480tF2lAvDAjpHppCETHWGPKEVMtUZMoUkvc5TYXuoTl26qKPtKI/asIA9B4/INTcs4LEX9gx61zKNTCEOmeoceUISMtU5MoU4/chTYQuqdRefqUXHHjNv26Jjj9G6i88c0B4hKxgWkA6ZQhwy1TnyhCRkqnNkCnH6kafCDvmrdePFde/RlV5eDAtIh0whDpnqXKs8SWSqzMhU5zhHIU4/8lTYgkqKwtUsLMwEUw5x/3meunSR9jQJEcMCWiNT5UamworLk0SmyoJMhcU5qtwGmafCDvlLQld68SWNl2VYQHhkqvjIVH+RqeIjU/1Dnopv0HkqZUFFV3rxJf3nedk5K3Tn5WdrxdJFMkkrli7SnZefzVWqLpCp4iNT/UWmio9M9Q95Kr5B56nQQ/7i0JVefK3+80waaoPOkaniI1P9RaaKj0z1D3kqvkHnqZQ9VHSlFx/Tp/YXmSo+MtVfZKr4yFT/kKfiG3SeSllQter6e+yFPTpvy1P6zfV/p/O2PMW6DznEf579RaaKj0z1F5kqPjLVP+Sp+Aadp1IO+ZOYCabo2pmSGGGRqWIjU/1HpoqNTPUXeSq2QeeptAVVnFY3tSFbktaVYPx5NpCpfCFT2Uem8qPV2kdkavDIU75k9RxFQdWAmWDyg6tK+UCm8oNM5QOZygfylA/kKT+ynKlS3kOVZNA3taF9rCuRD2QqP8hUPpCpfCBP+UCe8iPLmaKgajDom9rQPq4q5QOZyg8ylQ9kKh/IUz6Qp/zIcqYY8tdg0De14Whx42VZVyIfyFT2kKl8I1PZQp7yjTxlTx4zRUHVBDeJZkfSeNl1F585r03iqlJWkansIFPFQKaygTwVA3nKjrxmioKqQ61m7EE6ccc1abzs9vVfksRVpbwjU+ElHVMyVXxkKjzOUeVGpsIrWqYoqDqQ5dlF8izpuLYaL8tVpXwjU+G1OqZkqtjIVHico8qNTIU3kExNTkobNki7d0sjI9LmzVKlku4HaIJJKTrQanYRVtpOJ+m4MvtOsZGp8FodUzJVbGQqPM5R5UamwutZpiYnpZUrpQULoq+Tk3Pbx8el6WnJPfo6Pj7XHgAFVQeSquZatb3n4BG55qrt+mARuuaSjiuz7xRbN5kiT821urpHpoqNTIXHOarcyFR4PclUUtG0YYM0MzP/9TMz0fZAKKg6kFQ1t3MFo1XBVVZJx/Wyc1bozsvP1oqli2SSVixdpDsvP5tu9oJImynyFK/V1T0yVWxkKjzOUeVGpsLrKlNxvVBJRdPu3c13JG57ChRUHUiqmltdFc7yYmT9EnelptXViMvOWaHt67+kV7d8RdvXf4kTVYGkzRR5Sp8niUwVGZlKj3MUmiFT6aXO1MvPaPtffFOvfudSbf+Lb+qyl5+JXpTUC5VUNI2MNG+L254Ck1J0IGmtgq1PvpI4N36rgqvoM8i0c1NnkX9+NJc2U+SJPKE5MpUOmUIcMpVOq0yt+Mk2febu23Xywf16c+lyvXb9zfrcOZfMFU213qZa0SQl90KNjESvbVSbgKL+e0rS8HC0PRBz92DfrN/GxsZ8ampq0Lsh6ehfHCmqtmvdlOdteapp6FZUQ5T03ryELmk/k37+2jSYWWZmz7v72KD3o9fykqm4k1g7eap97zxnKu95ksqRqSzlSepdpvKSJ4lM5R2Zyp6fbr5bn/luQ2G04Vqdt+Upnbv9J7rhvz+oU99+S3uXnKTv/M6Vev68L2v7Z/Y1L3AmJqLiqFlhNDoa9TY1q1vMpIceiv+elUqqWf46yRQ9VIG0uoKVtBhZq27hPEzX2e1UzUCjVplKk6faiSrvmSJPSKMXmWp8X1bzJJEphFeaTMUUIz/dfLfO+vb1WvTh+5KkTx98U5/69vX6qaSx/7lLdz5xt4Y/itpOe3u/tjxxt26SpF/8Tbr7nZJ6oWrFUVzRVKkEnSa9ET1UfRR3xeE31/+dmv0rmKJu46SrZv2+ipH26l7er/6V4cqflK9MpcnTq1u+0vJ3MQ+ZkpTrPEnlyFSe8iQV4xyV9HOQqfwjU11kKqmHpkXbR1dfo6H35vbpo4WLNPTDe/XGtdfp0wffPOqvemPpyZIU2/bpQ/vje5riiqbR0fihe7VeqMBy2UNlZidKuk/SP5f0lqSb3P0/d/VNe7yIV6fiFiOLC0+rMbi9utIeF9Zuru4l9dChN4qeqTR5ktqbAjfrmfqPX/0seRqA4JnKUJ6k/JyjJDJVBEU/R0npM/X7O54+aqjc366+UI+9sEfP3fbneuSpBz5p+97/ukra9K3o74n7+ZPuS5LmF0zT09FzSapUNLPuRg2/N39fh947opl1N+rkg/ub/twnH9wvs+bH5JRD+9Pf79SqF2qAsjTL3w8kfSDpFEkVSfeY2erU363VIl5x0y4OQNJsJ91O1R63/kFcW9I0n90sxMbUsgORnUz1MW+tZg8qQqbI08CEy1Q7C02mzVTgvPXqHCWRqZLLzzkqcBbXXXym/uCVZ/XcPd/Q/7nrUj13zzf0B688q3UXn6k/fHW7tjxxt057e78WyD8ZKveHr27Xz7f8QLc9/v15bbc9/n39fMsPPulJqv/5P7r6mpbrMM2su3Fe75M0VzBJ0sJ9e5se7oX79urNpcubtr25dLksZgY9qxVBw8PzG+qLpomJqEfKLPpa3wNVqUi7dkmzs9HXDBRTUkaG/JnZYkkHJJ3l7r+qbntI0h53Xx/3vsSu35Ur03cZDuAKR7tX26S5Gxf/3SM/j+0yjrvaduflZ0tqPq631U2Ue6snr07/viKckPI2lCJTmZL6nrekIRFkKhvKkKnUedq16+grylJ7mUpqa5W3hLa4G8/T5unVLV9JfK9EpjqVp0zl7hwVOotS7DC6mXU3anjf0WtVzfzjFfp/hz/QaW8f3Sv0+pLlOnHxb8S+b9Ebe2VNPu+7mdylBU1SMyvTAp/V6586Ofbv3HfDpnn3UEnSkWOP0/++5bv63Mplmfus3alOMpWVguocSf/D3RfVbbte0hfd/dK49yUGa8GCXIzPbEfo8eBJbUknoyyOle+XPJ2opIxlSsrcxQ0yNXhlyFTqPM3OJn84lMLnTUr9gTPVLF/Ve2vj2iXFtpGp5vKUqdyco3bt6k0Wk9oSZrKbTSh+pPi2N5cuj72f6aNZjy2YTjv0pm796r/XDY/+2SeTS0jSzNBx+s7l1+nWR/5DbP4l5aJoSpLHguqfSvqxu3+6bts1kirufkHDa8cljUvSyMjIudPNfiGl5AAkTbuYFLraVcOM/HKkvTIoKdXJqJ3pqIsqTycqKWOZknpzcaNHPVtkqj+Kmqkgedq1K/nDoRQ+b1LwIu2nuw7EX73ecK2+den182YBk6IPajddEn0Yi2ubOu/LscXWuovP1HO3/bn+pP4eky9dpfNb3WNSAHnKVG7OUbOzvcliUltCTmc++ChV79XW37kyNk/Lhn8jsWCq3bcVm6kC6yRTWbmH6l1JSxq2LZH0TuML3X3C3cfcfWz58uZjNyUlj89MWjE5abrGdsa891HSmO+kseJJbUlj5RljnivZyVTavCUt4NejLJIpJGgrU0HyJKXPVNq8pW1LyOnn7v3uvGJKkhZ9+L4+d+93JUk3PffQvA9xkjT80fu66bmHEtu+9/HLuuvJ+feY3PXk3frexy/rspefaXr/yWUvP9O7e2wydE92juTjHFX/tdP3pm1L+DmGt96ljxbOP998tHCRhrfepR9ecrVmho6b1zYzdJx+eMnVmjrvy1p/ybV6fclyzcr0+pLlWl+9OPHZ9X+kTb/3x/PaNv3eH+uz6/9IUnRePH/Tt/TVm36k02/8W331ph+VopjqmLsP/CFpsaIbE8+o2/agpC1J7zv33HM90cMPu4+OuptFXx9+eG778LB79F9q9Bgennt9/fbaY3Q0uS1jtv3sdV+18e999MbHP3ms2vj3vu1nrye21d77hTv/wVfe+Lh/4c5/+GR7mUma8gxkpd1HpjKVNm9mzdtqf3dSFuP2swtkKqwyZCp1nmptaTLVi/Nb2pwmtbn7bEz7rFliW09+jl4c71b/xu38DnTw/1ieMpWbc1Q37+3V701M27afve5/etk6f23Jcv9Y5q8tWe5/etk6zlFd6CRTAw/VJzsi/UjSX1VDdp6kQ5JWJ72nZbCSpAldi5NDLz7EdSMpIISnM3k6UdUemclUUlsviq1WJ6pW+5qATIVThkx1lSf39B+2Q3+o7NVFyH4XcWkv0vSiSOvm3yNG3jKVi3NUt+8NWDC3g3NUWHktqE6U9Jikw5J2S/paq/d0fbKKE/dL3s1/nMi1vJ2oPGuZStLvnmSymgllyNRA8tRKVoq0bt6bpZ62bnrSA4+IyVumcnOOQmnlsqBK8+h7sNL+B4/cy9uJKu0jcyerXvQkD2C4II5WhkxlLk+90u8r/1nqaUtbpLl3NVyyGTIFhEVB1Utx/8G3858fH9RyqwwnKs/bySpNT7J798MFEUQZMpWrPOVNVnraurnQWvIeqjQPMoV+oqAaBIYZFVoZTlSetUyl1Spr3Q4X5KJIEGXIVCHyVCb9LNJatZfgHqo0DzKFfqKgGoRuPsQh88pwovKsZaobrT78pBkuyEWRoMqQqcLkCen1arKDJsgUEFYnmcrEwr5pJa6YPQhJCwYmLQw3O9vf/UTH8rRgYjcyl6leictq0sKQUm4W/c6DMmSqNHlCJpApIKw8LuxbDJVK9MFqdjb6Wv9hqtXCcSwMCPRPXFaTFobM0aLfAACgfyio+iXpgxofxoBsqFSkiYmo18ks+joxEW1PuiiyYYM0MzN/+8xMtF3iggkAAAVGQdUvSR/UWn0YA9A/9F4BAIAOUFD1U9wHtaQPYwCyoVe9VwAAINcoqLKA+6uAfAjdeyWRbwAAco6CKgu4vwrIt7S9V+QbAIDco6DKAu6vAvIvTe8V+QYAIPcoqLKC+6uAYkq6YMJwQAAAcm9o0DuAFkZGmi8mGjeMCED2VCrNF/lNyndtOGCtB6s2HLD2/QAAQCbQQ5V1ScOFJK5gA3nWzXBAsg8AQCZQUGVd0nAhbmgH8i3tcECyDwBAZlBQ5UHc/VXc0A7kX1y+WdsKAIBcoKDKMyasAIqrm7WtAABA31BQ5VmrBYEB5Ffata0k7q8CAKCPKKjyrNWEFQDyLc3aVtxfBQBAX1FQ5VnSFWyJq9RAUbEYOAAAmUFBlXdxV7C5Sg0UWzeLgXOxBQCAYCioioqr1EA5tXN/FRdbAAAIhoKqqJgFDCinVvdWcrEFAICgKKiKihkAgXJqdW9lq4stDAcEAKAjFFRFxQyAQHnF3V8lJV9sYTggAAAdo6AqKmYABNBM0sUWhgMCANAxCqoiYwZAAI2SLrZw7yUAAB2joCojrkID5RZ3saWdGQLp2QYAYB4KqjLiKjSAZpKGA9KzDQBAUxRUZcQMgACaSRoOSM82AABNUVCVETMAAogTNxyQnm0AAJqioCqjVjMAAkAj7q8CAKCpIAWVmV1rZlNm9r6ZPdCk/SIz22lmM2b2tJmN1rUdZ2b3m9nbZvaGmV0XYp/QQtI6NXwwGjgyhczJ+f1VZAoIi0wBc0L1UO2VdIek+xsbzOwkSY9KulnSiZKmJD1S95JbJZ0haVTShZJuMLNLAu0XOpWDD0YlQaaQLfm/v4pMAWGRKaAqSEHl7o+6+2OS/m+T5ssl7XD3H7v7e4pCtMbMVlXbr5R0u7sfcPdfSrpX0lUh9gsp5OODUeGRKWRSju+vIlNAWGQKmNOPe6hWS3qx9sTdD0v6taTVZrZM0qn17dU/r477ZmY2Xu1intq/f3+PdrnEcvDBCGQKGZP/mUODZYo8AZLIFEqmHwXV8ZIONWw7JOmEapsa2mttTbn7hLuPufvY8uXLg+4oVIQPRmVAppAt+Z85NFimyBMgiUyhZFoWVGb2jJl5zOO5Nv6OdyUtadi2RNI71TY1tNfaMAj5/2CUeWQKhTPgmUPJFBAWmQI607KgcvcL3N1iHue38XfskLSm9sTMFks6XdHY2gOS9tW3V/+8o7MfA8EwpXrPkSkUUtLMoT1GpoCwyBTQmVDTpg+Z2UJJx0g6xswWmtlQtXmbpLPMbG31NZskveTuO6vtD0raaGbLqjcrXiPpgRD7hZQG+MEIETIFhEWmgLDIFDAn1D1UGyUdkbRe0terf94oSe6+X9JaSZslHZD0eUlX1L33FkU3Kk5LelbSVnd/ItB+ITTWqOoXMgWERaaAsMgUUGXuPuh9SG1sbMynpqYGvRvlUVujqn5a9eHhUgwJNLPn3X1s0PvRa2QK/VKGTJEn9BOZAsLqJFP9mOUPRcEaVQAAAMA8FFRoH2tUAQAAAPNQUKF9rFEFAAAAzENBhfaxRhUAAAAwDwUV2scaVQAAAMA8Q61fAtSpVCigAAAAgCp6qBAOa1QBAACgZOihQhiNa1RNT0fPJXq0AAAAUFj0UCEM1qgCAABACVFQIQzWqAIAAEAJUVAhDNaoAgAAQAlRUCEM1qgCAABACVFQIQzWqAIAAEAJMcsfwmGNKgAAAJQMPVQAAAAAkBIFFfqHhX8BAABQMAz5Q3+w8C8AAAAKiB4q9AcL/wIAAKCAKKjQHyz8CwAAgAKioEJ/sPAvAAAACoiCCv3Bwr8AAAAoIAoq9AcL/wIAAKCAmOUP/cPCvwAAACgYeqgAAAAAICUKKmQDi/4CAAAghxjyh8Fj0V8AAADkFD1UGDwW/QUAAEBOUVBh8Fj0FwAAADlFQYXBY9FfAAAA5BQFFQaPRX8BAACQUxRUGDwW/QUAAEBOdV1QmdlxZnafmU2b2Ttm9oKZ/W7Day4ys51mNmNmT5vZaMP77zezt83sDTO7rtt9Qg5VKtKuXdLsbPS1xMUUmQLCIlNAWGQKmC9ED9WQpNckfVHSpyTdLOmvzWylJJnZSZIerW4/UdKUpEfq3n+rpDMkjUq6UNINZnZJgP0C8opMAWGRKSAsMgXU6bqgcvfD7n6ru+9y91l3f1zSq5LOrb7kckk73P3H7v6eohCtMbNV1fYrJd3u7gfc/ZeS7pV0Vbf7BeQVmQLCIlNAWGQKmC/4PVRmdoqk35K0o7pptaQXa+3ufljSryWtNrNlkk6tb6/+eXXo/UKOTU5KK1dKCxZEXycnB71HfUWmgLDIFBAWmULZBS2ozOxYSZOS/tLdd1Y3Hy/pUMNLD0k6odqmhvZaW9zfMW5mU2Y2tX///jA7juyanJTGx6Xpack9+jo+XpqiikwBYfU6U+QJZUOmgDYKKjN7xsw85vFc3esWSHpI0geSrq37Fu9KWtLwbZdIeqfapob2WltT7j7h7mPuPrZ8+fJWu4+827BBmpmZv21mJtqeU2QKCCtLmSJPKAIyBXSmZUHl7he4u8U8zpckMzNJ90k6RdJad/+w7lvskLSm9sTMFks6XdHY2gOS9tW3V/+8Q4Ak7d7d2fYcIFNAWGQKCItMAZ0JNeTvHkm/LelSdz/S0LZN0llmttbMFkraJOmlum7hByVtNLNl1ZsVr5H0QKD9Qt6NjHS2vTjIFBAWmQLCIlNAVYh1qEYl/StJn5X0hpm9W31UJMnd90taK2mzpAOSPi/pirpvcYuiGxWnJT0raau7P9HtfqEgNm+WhofnbxsejrYXFJkCwiJTQFhkCphvqNtv4O7TkqzFa/6bpFUxbe9L+mb1AcxXW+B3w4ZomN/ISFRMFXjhXzIFhEWmgLDIFDBf1wUV0HOVSqELKAAAAORX8HWoAAAAAKAsKKgAAAAAICUKKgAAAABIiYIK+TY5Ka1cKS1YEH2dnBz0HgEAAKBEmJQC+TU5KY2PSzMz0fPp6ei5xCQWAAAA6At6qJBfGzbMFVM1MzPRdgAAAKAPKKiQX7t3d7YdAAAACIyCCvk1MtLZdgAAACAwCirk1+bN0vDw/G3Dw9F2AAAAoA8oqJBflYo0MSGNjkpm0deJCSakAAAAQN8wyx/yrVKhgAIAAMDA0EMFAAAAAClRUAEAAABAShRUAAAAAJASBRWKa3JSWrlSWrAg+jo5Oeg9AgAAQMEwKQWKaXJSGh+XZmai59PT0XOJSSwAAAAQDD1UKKYNG+aKqZqZmWg7AAAAEAgFFYpp9+7OtgMAAAApUFChmEZGOtsOAAAApEBBhWLavFkaHp6/bXg42g4AAAAEQkGFYqpUpIkJaXRUMou+TkwwIQUAAACCYpY/FFelQgEFAACAnqKHCgAAAABSoqACAAAAgJQoqAAAAAAgJQoqAAAAAEiJggrlNDkprVwpLVgQfZ2cHPQeAQAAIIeY5Q/lMzkpjY9LMzPR8+np6LnErIAAAADoCD1UKJ8NG+aKqZqZmWg7AAAA0AEKKpTP7t2dbQcAAABiBCmozOxhM9tnZm+b2a/M7OqG9ovMbKeZzZjZ02Y2Wtd2nJndX33vG2Z2XYh9AmKNjHS2fQDIFBAWmQLCIlPAnFA9VHdKWunuSyT9vqQ7zOxcSTKzkyQ9KulmSSdKmpL0SN17b5V0hqRRSRdKusHMLgm0X8DRNm+WhofnbxsejrZnB5kCwiJTQFhkCqgKUlC5+w53f7/2tPo4vfr8ckk73P3H7v6eohCtMbNV1fYrJd3u7gfc/ZeS7pV0VYj9ApqqVKSJCWl0VDKLvk5MZGpCCjIFhEWmgLDIFDAn2D1UZvafzGxG0k5J+yT9pNq0WtKLtde5+2FJv5a02syWSTq1vr3659Wh9gtoqlKRdu2SZmejrxkqpmrIFBAWmQLCIlNAJNi06e7+b8zs30r6J5IukFS7anG8pP0NLz8k6YRqW+15Y1tTZjYuqTrHtd41s1fa2L2TJL3VxuvKiGMTr/7YjCa9sBfIVG5xbOIVPlPkqSc4PvHIVHP8zsTj2CRLlamWBZWZPSPpizHN2939/NoTd/9Y0nNm9nVJ/1rS9yW9K2lJw/uWSHqn2lZ7/l5DW1PuPiFpotV+N/wMU+4+1sl7yoJjE69Xx4ZMFRvHJl4ZMkWewuP4xCNTsfvP70wMjk2ytMen5ZA/d7/A3S3mcX7M24Y0N452h6Q1dTu6uNq2w90PKOoiXlP33jXV9wCFRKaAsMgUEBaZAjrT9T1UZnaymV1hZseb2TFmdrGkfynpqepLtkk6y8zWmtlCSZskveTuO6vtD0raaGbLqjcrXiPpgW73C8grMgWERaaAsMgUMF+ISSlcURfv65IOSPqupD9x9/8iSe6+X9JaSZur7Z+XdEXd+29RdKPitKRnJW119ycC7Fe9jrqKS4ZjE29Qx4ZM5RvHJh6Zao7fmWQcn3hkqjl+Z+JxbJKlOj7m7qF3BAAAAABKIdi06QAAAABQNhRUAAAAAJBSoQsqMzvRzLaZ2WEzmzazrw16nwbFzK41sykze9/MHmhou8jMdprZjJk9bWZ9X8tikMzsODO7r/o78o6ZvWBmv1vXXurjU49MzSFTzZGn9pGnOeQpHplqH5maQ6bi9SJThS6oJP1A0geSTpFUkXSPmZV1Je69ku6QdH/9RjM7SdKjkm6WdKKkKUmP9H3vBmtI0muK1tz4lKJj8ddmtpLjcxQyNYdMNUee2kee5pCneGSqfWRqDpmKFzxThZ2UwqI1Dw5IOsvdf1Xd9pCkPe6+fqA7N0Bmdoek09z9qurzcUlXufsXqs8XK1oh+py66U1Lx8xekvRtSf9IHB9JZCoOmWqNPB2NPDVHntpDpo5GppojU+3pNlNF7qH6LUkf10JV9aKksl6piLNa0XGRJLn7YUVTmZb2OJnZKZhPqP0AAAH6SURBVIp+f3aI41OPTLWH35k65CkWeWoPvzMNyFQsMtUefmcahMhUkQuq4yUdath2SNIJA9iXLOM41TGzYyVNSvrL6pUIjs8cjkV7OE5V5CkRx6I9HKc6ZCoRx6I9HKc6oTJV5ILqXUlLGrYtkfTOAPYlyzhOVWa2QNJDisZfX1vdzPGZw7FoD8dJ5KkNHIv2cJyqyFRLHIv2cJyqQmaqyAXVryQNmdkZddvWKOrOw5wdio6LpE/Gip6ukh0nMzNJ9ym6kXWtu39YbeL4zCFT7Sn97wx5agt5ag+/MyJTbSJT7eF3RuEzVdiCqjrm8VFJt5nZYjM7T9K/UFSJlo6ZDZnZQknHSDrGzBaa2ZCkbZLOMrO11fZNkl4q4Y2J90j6bUmXuvuRuu0cnyoyNR+ZSkSeWiBP85GnlshUC2RqPjLVUthMuXthH4qmO3xM0mFJuyV9bdD7NMBjcaskb3jcWm37Z5J2Sjoi6RlJKwe9v30+NqPV4/Geoq7e2qPC8TnqWJGpuWNBppofF/LU/rEiT3PHgjzFHxsy1f6xIlNzx4JMxR+b4Jkq7LTpAAAAANBrhR3yBwAAAAC9RkEFAAAAAClRUAEAAABAShRUAAAAAJASBRUAAAAApERBBQAAAAApUVABAAAAQEoUVAAAAACQEgUVAAAAAKT0/wFx0txpoz32QAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x216 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "_,axs = plt.subplots(1,4,figsize=(12,3))\n",
    "for ax in axs: show_preds(apply_step(params, False), ax)\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 7: stop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summarizing Gradient Descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: G Pages: 1 -->\n",
       "<svg width=\"591pt\" height=\"78pt\"\n",
       " viewBox=\"0.00 0.00 591.49 78.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 74)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-74 587.4867,-74 587.4867,4 -4,4\"/>\n",
       "<!-- init -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>init</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"27\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"27\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">init</text>\n",
       "</g>\n",
       "<!-- predict -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>predict</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"126.0969\" cy=\"-18\" rx=\"35.194\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"126.0969\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">predict</text>\n",
       "</g>\n",
       "<!-- init&#45;&gt;predict -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>init&#45;&gt;predict</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M54.0787,-18C62.3227,-18 71.6196,-18 80.7269,-18\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"80.8626,-21.5001 90.8626,-18 80.8625,-14.5001 80.8626,-21.5001\"/>\n",
       "</g>\n",
       "<!-- loss -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>loss</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"225.1938\" cy=\"-52\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"225.1938\" y=\"-48.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">loss</text>\n",
       "</g>\n",
       "<!-- predict&#45;&gt;loss -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>predict&#45;&gt;loss</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M155.2932,-28.0172C166.6224,-31.9043 179.6698,-36.3808 191.4018,-40.406\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"190.2859,-43.7234 200.8806,-43.6582 192.5577,-37.1023 190.2859,-43.7234\"/>\n",
       "</g>\n",
       "<!-- gradient -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>gradient</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"361.8403\" cy=\"-52\" rx=\"39.7935\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"361.8403\" y=\"-48.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">gradient</text>\n",
       "</g>\n",
       "<!-- loss&#45;&gt;gradient -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>loss&#45;&gt;gradient</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M252.5178,-52C269.4967,-52 291.836,-52 311.8929,-52\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"312.1329,-55.5001 322.1329,-52 312.1328,-48.5001 312.1329,-55.5001\"/>\n",
       "</g>\n",
       "<!-- step -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>step</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"465.4867\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"465.4867\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">step</text>\n",
       "</g>\n",
       "<!-- gradient&#45;&gt;step -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>gradient&#45;&gt;step</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M394.0665,-41.4286C405.9515,-37.5298 419.4492,-33.1021 431.4862,-29.1535\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"432.7754,-32.4142 441.1862,-25.9715 430.5935,-25.7629 432.7754,-32.4142\"/>\n",
       "</g>\n",
       "<!-- step&#45;&gt;predict -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>step&#45;&gt;predict</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M438.4132,-18C380.3272,-18 243.2155,-18 171.5401,-18\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"171.4571,-14.5001 161.4571,-18 171.4571,-21.5001 171.4571,-14.5001\"/>\n",
       "<text text-anchor=\"middle\" x=\"287.1938\" y=\"-21.8\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">repeat</text>\n",
       "</g>\n",
       "<!-- stop -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>stop</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"556.4867\" cy=\"-18\" rx=\"27\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"556.4867\" y=\"-14.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">stop</text>\n",
       "</g>\n",
       "<!-- step&#45;&gt;stop -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>step&#45;&gt;stop</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M492.7897,-18C501.068,-18 510.3085,-18 519.1272,-18\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"519.203,-21.5001 529.203,-18 519.203,-14.5001 519.203,-21.5001\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x7f10d352e250>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gv('''\n",
    "init->predict->loss->gradient->step->stop\n",
    "step->predict[label=repeat]\n",
    "''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The MNIST Loss Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_x = torch.cat([stacked_threes, stacked_sevens]).view(-1, 28*28)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([12396, 784]), torch.Size([12396, 1]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_y = tensor([1]*len(threes) + [0]*len(sevens)).unsqueeze(1)\n",
    "train_x.shape,train_y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([784]), tensor([1]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dset = list(zip(train_x,train_y))\n",
    "x,y = dset[0]\n",
    "x.shape,y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_x = torch.cat([valid_3_tens, valid_7_tens]).view(-1, 28*28)\n",
    "valid_y = tensor([1]*len(valid_3_tens) + [0]*len(valid_7_tens)).unsqueeze(1)\n",
    "valid_dset = list(zip(valid_x,valid_y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def init_params(size, std=1.0): return (torch.randn(size)*std).requires_grad_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = init_params((28*28,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bias = init_params(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([20.2336], grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(train_x[0]*weights.T).sum() + bias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[20.2336],\n",
       "        [17.0644],\n",
       "        [15.2384],\n",
       "        ...,\n",
       "        [18.3804],\n",
       "        [23.8567],\n",
       "        [28.6816]], grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def linear1(xb): return xb@weights + bias\n",
    "preds = linear1(train_x)\n",
    "preds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ True],\n",
       "        [ True],\n",
       "        [ True],\n",
       "        ...,\n",
       "        [False],\n",
       "        [False],\n",
       "        [False]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corrects = (preds>0.0).float() == train_y\n",
    "corrects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4912068545818329"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corrects.float().mean().item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights[0] *= 1.0001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4912068545818329"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds = linear1(train_x)\n",
    "((preds>0.0).float() == train_y).float().mean().item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trgts  = tensor([1,0,1])\n",
    "prds   = tensor([0.9, 0.4, 0.2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mnist_loss(predictions, targets):\n",
    "    return torch.where(targets==1, 1-predictions, predictions).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.1000, 0.4000, 0.8000])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.where(trgts==1, 1-prds, prds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.4333)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnist_loss(prds,trgts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.2333)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnist_loss(tensor([0.9, 0.4, 0.8]),trgts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sigmoid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(x): return 1/(1+torch.exp(-x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEMCAYAAAA/Jfb8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXyU5b3+8c8XCCQkJGwh7DvIpoBEEBStVetyatViq1VxrxW1bqeeWv1prdra2uOx1VqXUxTFXStuVG3VqrhUWQOEfd9JIGTfk+/vjwk9MSZmgCTPzOR6v17zknnmnuEyzFw83M8z92PujoiIxJY2QQcQEZGmp3IXEYlBKncRkRikchcRiUEqdxGRGKRyFxGJQSp3iTlmdqeZrQs6x35mNsvM3mtkzCVmVtlSmST2qdwlqphZgpndbWZrzazEzPaa2Xwzu67WsP8Gjg4qYz2uB34QdAhpXdoFHUDkAD0CnECoMDOAZGA80H//AHcvBAoDSVcPd88LOoO0Ptpzl2hzFvB7d3/N3Te6e4a7z3L3u/YPqG9axsxuMLNtZlZsZu+a2XQzczPrW/P4JWZWaWYnmNmymn8VfGRmvc3sODNbbGZFZvaemfWp89oXm9kKMyur+T3uMbN2tR7/yrSMhdxtZllmVmhmLwBdmusHJq2Tyl2izU7gVDPrGu4TzOz7hKZqfg+MBZ4HflfP0DbAL4ErgGOA3sCLwF3ADOBYoC/wP7Ve+z+AJ4DZwOHAfwLX1LxOQ64DbgJuBo4EFjUyXuTAubtuukXNjVDpbgaqgKXA48CZgNUacyewrtb9T4HZdV7nt4ADfWvuX1Jzf1ytMTfXbJtQa9uNwJ5a9+cBL9V57euBEqB9zf1ZwHu1Ht8G/LrOc14BKoP++eoWOzftuUtUcfdPgSHAVOApIA34K/CGmVkDTxsF/KvOts/re3lgWa37u2r+u7TOtm5m1rbm/mjg4zqv8xEQX5PzK8wsGegDfFbnoU8ayC5yUFTuEnXcvdLdP3P3+939TEJ73d8Fjvump4Xx0tXuXlX3Oe5eUc/rWD3bqPNYfb/nNz0m0mRU7hILVtb8t0cDj68AJtfZ1lSnSmYCx9fZdhyhaZkNdQd76MyZ7YSml2qre1/kkOhUSIkqZvYRoQOiC4BsYCjwGyAX+GcDT7sfeNHMvgTeBqYAF9U8dqh70PcCb5rZLcCrwDhCc/73u3v5N+S528xWEZou+h5w0iHmEPkK7blLtHkbuAD4G7AaeBJYCxzj7nvqe4K7vwr8F3ALoTn1C4Bf1Txceihh3P1vwGXAxcBy4AHgz7Vevz5/BB6sGbuE0L8q7vqG8SIHzNw19Setj5ndAVzv7t2CziLSHDQtIzHPzOIInX/+N6CI0DdcbwYeDjKXSHPSnrvEvJpvi74FTAA6ARuBpwl901WLdUlMUrmLiMQgHVAVEYlBETHn3r17dx84cGDQMUREosrChQv3uHtqfY9FRLkPHDiQBQsWBB1DRCSqmNnmhh7TtIyISAwKq9zN7FozW1CzXvWsRsbeaGa7zCzPzJ4wsw5NklRERMIW7p77DuAeQutWN8jMTiH0LcATgYHAYL75m3oiItIMwip3d3/V3V8D9jYy9GJgprtnuvs+4G5CK/aJiEgLauo599GErmu5XwaQZmb6ireISAtq6nJPAmpfDHj/rzvVHWhmV9bM4y/Izs5u4hgiIq1bU5d7IaGr0e+3/9cFdQe6++Punu7u6amp9Z6mKSIiB6mpz3PPJHQB4pdq7o8Fdrt7Y3P1IiIxzd3JKSpnV34pWfllZBWUsju/jPH9OzN1WNPv4IZV7jULL7UD2gJtzSye0MV86y669DQwy8yeJXSV+v9H6OLAIiIxrbyymu25JWzbV8y2fSVs31fCjtwStueWsDOvlF35pZRXVn/teTO+NSS4cidU0r+sdf9C4Fdm9gShS5iNcvct7v6Omd1H6Io4CYQuXPzLr72aiEgUqqiqZktOMRuyi9i4p5CNe4rZtKeILTnF7MwrobrWOoxt2xg9k+Pp3Tmecf0606tzPD2TQ7ceyfH06NSB1E4diI9r2/BveAgiYlXI9PR01/IDIhIpqqqdjXsKWbWrgDW7C1m7u4C1WYVs3ltERdX/dWaXjnEM7J7IgK4d6d8tkf5dO9KvSwJ9u3YkrVMH2rVt3kUAzGyhu6fX91hErC0jIhKU0ooqVu8qYNn2PDJ35JG5I5/Vuwooq5lCaWMwoFsiQ3skcfKoNIamJjE4NZHB3ZNI6RgXcPqGqdxFpNVwd7bkFLNw8z4Wb8klY1suK3fm/3tvPCUhjtG9k5l+9ABG9kpmRK9ODElNarapk+akcheRmFVd7azclc8XG3L4cmMOCzbvY09hGQCJ7dtyRN/OXDF1MEf0SWFMnxT6dknAzAJO3TRU7iISUzbtKeKTdXv4dN0ePlu/l7ySCgD6dklg6rDuTBjQhfSBXRjWoxNt28RGkddH5S4iUa20oorP1+/lw9VZfLgmm817iwHonRLPd0alMXlINyYN7kafzgkBJ21ZKncRiTq5xeX8Y8Vu3lu5m4/X7KGkoor4uDZMGdKdy48dxNRhqQzs1jFmplgOhspdRKLCvqJy3sncxd+W7eTz9XuprHZ6pcRzzoS+nDiyB0cP7haVBz6bi8pdRCJWSXkVf1+xizeW7OCjNdlUVjsDunXkx8cN5rQxPTm8T0qr3jv/Jip3EYko7s6iLft4ZeE23srYSUFZJT2T47ns2EF8b2xvRvdOVqGHQeUuIhEhr7iCvy7axnNfbmFdViEJcW05/fBeTJvQh6MHdaNNDJ/Z0hxU7iISqBU78pn12UZeX7KDsspqxvbrzH3TjuD0I3qR1EEVdbD0kxORFldd7by3cjczP9nIFxtziI9rw/eP7MuFR/dndO+UoOPFBJW7iLSYssoqXlu8ncc+3sCG7CL6dE7g1tNHcG56/4hepyUaqdxFpNmVVlTxwpdbePSjDezKL2V072Qe/NF4Th/Ts9lXTmytVO4i0mxKK6p49ostPPrRerILypg4qCu//8ERHDu0u854aWYqdxFpcpVV1byycBt/fH8tO/NKmTKkGw/9aDxHD+4WdLRWQ+UuIk3G3XlvZRb3vr2SDdlFjOvXmft/MJYpQ7sHHa3VUbmLSJNYvj2Pu99awRcbcxicmsjj0ydw8qg0Tb8EROUuIockp6ic37+7mhfmb6FLx/bcfeZozpvYnzgdKA2Uyl1EDkp1tfPcl1v4/burKSyr5NIpg7jh5GEkx+uUxkigcheRA7ZqVz6/eHUZi7fkMnlwN3515miGp3UKOpbUonIXkbCVVlTx4PtrefzjDSQnxPHAuWM5a1wfzatHIJW7iIRl8ZZ93PzKUtZlFXLOhL7cdvpIuiS2DzqWNEDlLiLfqKyyigf+sZbHP15PWnI8T102keOHpwYdSxqhcheRBq3ZXcD1Lyxh5c58zk3vx23fHakDplFC5S4iX+PuPPXZJu59exVJHdrxl4vSOWlUWtCx5ACo3EXkK3KLy/nZy0t5b+VuTjgslfvOGUtqpw5Bx5IDpHIXkX9buHkf1z2/mKyCUm7/7iguO2agzoSJUip3EcHdefLTTfzmbyvp1TmeV66awth+nYOOJYdA5S7SyhWXV/KLV5fx+pIdnDQyjft/OJaUBB00jXYqd5FWbMveYq6cvYDVuwu4+ZTDmHH8EF2IOkaEtbKPmXU1szlmVmRmm83s/AbGdTCzR81st5nlmNmbZtanaSOLSFP4fP1eznz4E3bmlTLr0olcc8JQFXsMCXfZtoeBciANuAB4xMxG1zPuemAycATQG8gFHmqCnCLShJ77YgvTZ35Bt6QOvH7NMfpSUgxqtNzNLBGYBtzu7oXu/gnwBjC9nuGDgHfdfbe7lwIvAPX9JSAiAaiqdu5+awW3zlnGscO68+rVUxjYPTHoWNIMwplzHw5UufuaWtsygOPrGTsT+KOZ7d9rvwB4+5BTisghKymv4oYXF/Nu5m4umTKQ2787iraaholZ4ZR7EpBXZ1seUN/6nmuALcB2oApYBlxb34ua2ZXAlQD9+/cPM66IHIy9hWVc/tQCMrblcsd3R3HZsYOCjiTNLJw590Iguc62ZKCgnrGPAPFANyAReJUG9tzd/XF3T3f39NRUzfeJNJetOcWc8+jnrNqVz6MXTlCxtxLhlPsaoJ2ZDau1bSyQWc/YscAsd89x9zJCB1MnmpmujisSgJU785n2yGfkFJXz7BWTOGV0z6AjSQtptNzdvYjQHvhdZpZoZscAZwKz6xk+H7jIzFLMLA64Gtjh7nuaMrSING7+phx++NjntDHj5asmM2FA16AjSQsK91TIq4EEIAt4Hpjh7plmNtXMCmuN+xlQCqwFsoHTgbObMK+IhGHe2mymz/yC1KQOvDJjsi6B1wqF9Q1Vd88Bzqpn+zxCB1z3399L6AwZEQnI3zN3ce1zixmcmsjsyydpRcdWSssPiMSQNzN2cMOLSxjTJ4WnLj2Kzh11GbzWSuUuEiNeX7KdG19cQvqArsy8JJ1OumJSq6ZyF4kBry3ezk0vLeGogV154pKjSOygj3Zrp3eASJTbX+wTB4WKvWN7faxF5S4S1eYu3clNLy1h0qBuPHHJUSS0bxt0JIkQ4Z4KKSIR5u+Zu7j+hcUc2b8LMy9JV7HLV6jcRaLQR2uyufa5xYzuk8KTl2oqRr5O5S4SZRZsyuEnsxcwpEcST186UWfFSL1U7iJRZMWOfC6dNZ/eKQnMvnwiKR1V7FI/lbtIlNi4p4iLnviSpA7tmH3FJLon6Zun0jCVu0gUyMovZfrML6h2Z/blk+jTOSHoSBLhVO4iES6/tIKLn5xPTlE5sy49iqE9khp/krR6KneRCFZWWcVVsxeydncBj144gSP6dg46kkQJnT8lEqGqq52fvbyUz9bv5YFzx3LccF2xTMKnPXeRCPW7d1fxZsYObjltBGeP7xt0HIkyKneRCPTMvzbz2EcbuPDo/vzkuMFBx5EopHIXiTAfrNrNHa8v59sjenDnGaMxs6AjSRRSuYtEkBU78rn2ucWM6p3MQz8aT7u2+ojKwdE7RyRCZOWXcsVT80lJiGPmxVqTXQ6N3j0iEaCkvIofP72A3JIKXr5qMmnJ8UFHkiinchcJWOiUxwyWbs/jsQsnMLp3StCRJAZoWkYkYA9+sJa5y3Zyy6kj+M7onkHHkRihchcJ0NvLdvKH99Yy7ci+XKlTHqUJqdxFApK5I4+bXspgfP/O/PrsMTrlUZqUyl0kAHsKy7jy6YV07hjHY9MnEB+nS+RJ09IBVZEWVlFVzTXPLmJPYRmvXDWFHp10Zow0PZW7SAv79dyVfLExhwfOHcvhfXVmjDQPTcuItKCXF2xl1mebuPzYQVoMTJqVyl2khSzdlsttry1nypBu/OK0EUHHkRinchdpAXsLy7hq9kJSkzrwp/OP1Jox0uw05y7SzCqrqrnuhcXsKSrnr1dNoWti+6AjSSsQ1u6DmXU1szlmVmRmm83s/G8Ye6SZfWxmhWa228yub7q4ItHnv/++hk/X7eWes8boAKq0mHD33B8GyoE0YBww18wy3D2z9iAz6w68A9wIvAK0B3TUSFqtd5bv4tGP1nP+pP78ML1f0HGkFWl0z93MEoFpwO3uXujunwBvANPrGX4T8K67P+vuZe5e4O4rmzaySHTYkF3Iz17OYGy/zvzyjFFBx5FWJpxpmeFAlbuvqbUtAxhdz9ijgRwz+8zMsszsTTPr3xRBRaJJcXklM55ZRFxb488XHEmHdvoGqrSscMo9Ccirsy0P6FTP2L7AxcD1QH9gI/B8fS9qZlea2QIzW5CdnR1+YpEI5+7cNmc5a7IK+MN54+nTOSHoSNIKhVPuhUBynW3JQEE9Y0uAOe4+391LgV8BU8zsa0eR3P1xd0939/TU1NQDzS0SsZ79YgtzFm/nhhOHc/xwvbclGOGU+xqgnZkNq7VtLJBZz9ilgNe6v//XWu5OWoVl2/K4680VHDc8lZ9+e2jQcaQVa7Tc3b0IeBW4y8wSzewY4Exgdj3DnwTONrNxZhYH3A584u65TRlaJBLlFVdw9XML6ZbUnj+cO442bbRPI8EJ92tyVwMJQBahOfQZ7p5pZlPNrHD/IHf/ALgVmFszdijQ4DnxIrHC3fnZKxnszC3lT+cfqS8qSeDCOs/d3XOAs+rZPo/QAdfa2x4BHmmSdCJR4i/zNvKPFbu5/bujmDCgS9BxRLS2jMihWrh5H797ZxWnju7JZccMDDqOCKByFzkk+4rK+elzi+jVOZ7fnXOELpUnEUMLh4kcpOpq5z9fzmBPYTl/nTGFlIS4oCOJ/Jv23EUO0v/O28AHq7K47T9GakEwiTgqd5GDsHBzDve9u5rTD+/JRZMHBB1H5GtU7iIHKDTPvpg+nRP47TTNs0tk0py7yAFwd35Wa549OV7z7BKZtOcucgD+Mm8j76/K4tbTR2ieXSKayl0kTIu3hM5nP2V0GhdPGRh0HJFvpHIXCUNecQXXPreYninx3HfOWM2zS8TTnLtII9ydn/91KbvzS3n5qsk6n12igvbcRRrx9OebeSdzFz8/dQTj+2vdGIkOKneRb7B8ex6/nruSb4/owRVTBwUdRyRsKneRBhSUVnDtc4voltSe+3+geXaJLppzF6mHu3PrnOVs3VfCC1ceTRetzy5RRnvuIvV4cf5W3szYwU0nD+eogV2DjiNywFTuInWs3lXAL9/I5Nih3Zlx/JCg44gcFJW7SC3F5ZVc89wiOsXH8YCugypRTHPuIrXc8Xom67MLeebySaR26hB0HJGDpj13kRp/XbiNVxZu46ffHsYxQ7sHHUfkkKjcRYB1WQX8v9eWM3FQV64/cVjQcUQOmcpdWr2S8iqueXYxCe3b8uB542mreXaJAZpzl1bvzjcyWb27gKcum0jPlPig44g0Ce25S6v22uLtvLhgK1d/awjHD08NOo5Ik1G5S6u1LquQW+cs46iBXbjp5OFBxxFpUip3aZVC8+yLiI9ry4M/Gk+7tvooSGzRnLu0Svvn2WddehS9UhKCjiPS5LS7Iq3Oq4u28eKCrVxzwhC+dViPoOOINAuVu7Qqa3cXcNuc0PnsN56keXaJXSp3aTWKyiqZ8ewiEju05U+aZ5cYpzl3aRXcndvmLGNDzboxPZJ1PrvEtrB2Xcysq5nNMbMiM9tsZuc3Mr69ma0ys21NE1Pk0Dz35RZeW7KDG08azhStGyOtQLh77g8D5UAaMA6Ya2YZ7p7ZwPibgSwg6dAjihyapdty+dUbKzh+eCrXnDA06DgiLaLRPXczSwSmAbe7e6G7fwK8AUxvYPwg4ELg3qYMKnIwcovLmfHMIlI7ddD67NKqhDMtMxyocvc1tbZlAKMbGP8QcCtQcojZRA5JdbVzw4tLyC4o488XHElXXQdVWpFwyj0JyKuzLQ/oVHegmZ0NtHP3OY29qJldaWYLzGxBdnZ2WGFFDsRDH6zjw9XZ3HHGKMb26xx0HJEWFU65FwLJdbYlAwW1N9RM39wH/DSc39jdH3f3dHdPT03Vgk3StD5cncUf3l/D2eP7cMGk/kHHEWlx4RxQXQO0M7Nh7r62ZttYoO7B1GHAQGCemQG0B1LMbBdwtLtvapLEIo3YsreY619YwmFpnfjN2YdT834UaVUaLXd3LzKzV4G7zOwKQmfLnAlMqTN0OdCv1v0pwJ+AIwHNu0iLKCmv4qpnFuLuPDZ9Agnt2wYdSSQQ4X5F72oggdDpjc8DM9w908ymmlkhgLtXuvuu/TcgB6iuuV/VLOlFanF3bnttGSt35fPH88YzoFti0JFEAhPWee7ungOcVc/2eTRwLru7fwj0PZRwIgfi6c838+qi7dxw0jBOGKEFwaR10+IaEhM+X7+Xu95awUkj07ju27rAtYjKXaLe9twSrnluEQO7deSBc8fqi0oiqNwlypVWVPGT2QuoqKzm8YvS6RQfF3QkkYigVSElark7N7+ylMwd+fzlonSGpGopI5H9tOcuUevPH67nzYwd3HzKYZw4Mi3oOCIRReUuUenvmbv4/burOXNcb2YcPyToOCIRR+UuUWfVrnxufHEJY/um8LtpR+gbqCL1ULlLVMkuKOPyWQtIim/HY9PTiY/TN1BF6qMDqhI1SiuquHL2AnKKynn5qsn0TNGl8kQaonKXqLD/zJjFW3J59MIJjOmTEnQkkYimaRmJCg/8Yw1vZuzg56eO4NQxPYOOIxLxVO4S8V6av5UHP1jHuen9uOr4wUHHEYkKKneJaPPWZnPrnGUcNzyVe84eozNjRMKkcpeItXJnPjOeWcTQHkk8fP544trq7SoSLn1aJCJt21fMJU9+SVKHdjx56VFaM0bkAKncJeLsKyrnoie+pKS8iqcvn0ivlISgI4lEHZ0KKRGlpLyKy56az7Z9JTxz+SSGp3UKOpJIVNKeu0SM8spqrn52IRlbc3nwvPFMHNQ16EgiUUt77hIRqqqd/3w5g3+uzube7x+uc9lFDpH23CVw7s4dry/nzYwd3HLaCH40sX/QkUSinspdAuXu3Pfuap79YgtXHT+Eq7R8r0iTULlLoB58fx2PfLie8yf15+enHhZ0HJGYoXKXwDz20XoeeG8N50zoyz1n6tunIk1J5S6BePLTjdz79irOGNub3007gjZtVOwiTUlny0iLe+KTjdz11gpOHd2T//nhWNqq2EWanMpdWtRf5m3gnrkrOXV0Tx7SejEizUafLGkx+4v9tDEqdpHmpj13aXbuzkMfrON//rGG/zi8F384b5yKXaSZqdylWbk7v31nFY99tIFpR/bld9MOp52KXaTZqdyl2VRVO798YznP/GsL048ewK++N1pnxYi0EJW7NIuyyipuejGDuct28pPjB3PLqSN0HrtICwrr38dm1tXM5phZkZltNrPzGxh3s5ktN7MCM9toZjc3bVyJBoVllVw2az5zl+3kttNH8ovTRqrYRVpYuHvuDwPlQBowDphrZhnunllnnAEXAUuBIcDfzWyru7/QVIElsmXll3LZU/NZubOA+38wlmkT+gYdSaRVanTP3cwSgWnA7e5e6O6fAG8A0+uOdff73H2Ru1e6+2rgdeCYpg4tkWnN7gLO/vNnbMgu4i8XpavYRQIUzrTMcKDK3dfU2pYBjP6mJ1no3+FTgbp79xKDPl23h2l//ozyqmpe+slkThjRI+hIIq1aOOWeBOTV2ZYHNHb9sztrXv/J+h40syvNbIGZLcjOzg4jhkSqZ7/YzMVPfEmvzvG8ds0xjOmTEnQkkVYvnDn3QiC5zrZkoKChJ5jZtYTm3qe6e1l9Y9z9ceBxgPT0dA8rrUSUyqpq7n5rBU99vpnjh6fy0PnjSY6PCzqWiBBeua8B2pnZMHdfW7NtLA1Mt5jZZcAtwHHuvq1pYkqkySkq57rnF/PJuj38eOogbjltpBYAE4kgjZa7uxeZ2avAXWZ2BaGzZc4EptQda2YXAL8BTnD3DU0dViLDsm15XPXMQrILy7jvnCP4YXq/oCOJSB3hfg/8aiAByAKeB2a4e6aZTTWzwlrj7gG6AfPNrLDm9mjTRpYgvbRgK9Me/QyAV66arGIXiVBhnefu7jnAWfVsn0fogOv++4OaLppEkuLySu54PZNXFm7j2KHdefBH4+ma2D7oWCLSAC0/II1avauAa55bxPrsQq47cRjXnzhM8+siEU7lLg1yd575Ygu/nruCpA5xPHP5JI4Z2j3oWCISBpW71Cu7oIyf/3UpH6zK4rjhqfz3D46gR6f4oGOJSJhU7vI17yzfxW1zllFQVsmdZ4zioskDtVSvSJRRucu/5RSV88s3MnkzYwejeyfz/LnjGJ7W2BeRRSQSqdwFd2fusp3c+UYmeSUV3HTycGZ8a4guhScSxVTurdzWnGLueH05/1ydzeF9Uph9+SRG9qq72oSIRBuVeytVVlnFzE828tD76zCD2787iosnD9D1TUVihMq9FfpwdRa/enMFG/cUcfKoNO783mj6dE4IOpaINCGVeyuydncB9769ig9WZTG4eyJPXTaR44enBh1LRJqByr0VyC4o4w/vreGF+Vvp2L4tvzhtBJceM4j27TQFIxKrVO4xLK+4gsfnreeJTzZRUVXN9KMHcN2Jw7QmjEgroHKPQfmlFTz16Sb+d94G8ksr+d7Y3tx48nAGdU8MOpqItBCVewzJLS7nyU838cSnGykoreSkkT246eTDGNVbpzaKtDYq9xiwPbeEmfM28sL8LRSXV3HK6DR++u1hupapSCumco9iS7bm8uSnG3lr6U4MOGNsb648brC+hCQiKvdoU1pRxTvLdzHrs00s2ZpLUod2XDx5IJdPHaRz1UXk31TuUWJDdiHPf7mFVxZuY19xBYO6J3LnGaM4J70fSR30xygiX6VWiGD5pRXMXbqTVxZuY+HmfbRrY5w8Ko0LJg1gypBuWoZXRBqkco8wpRVVfLg6mzcytvP+yizKKqsZ2iOJW04bwffH96FHsi6YISKNU7lHgNKKKj5ek83by3fx3srdFJRW0j2pPecd1Y+zxvdhXL/OmGkvXUTCp3IPSE5ROf9clcV7K3fz8ZpsisqrSEmI45TRPfne2N5MGdJNKzSKyEFTubeQqmpn+fY8PlydzUdrsliyNZdqh7TkDnxvXB9OG9OTyUO66QIZItIkVO7NxN1Zn13Evzbs5dN1e/hs/V7ySiowgyP6pHDtt4dx0sgejOmdogOjItLkVO5NpKKqmpU781mwaR8LNufw5cYc9hSWA9A7JZ7vjErj2GHdOXZod7oldQg4rYjEOpX7QXB3tu0rYdn2PJZszWXJ1lyWbcujpKIKCJX51GGpTBrUlUmDuzGwW0cdEBWRFqVyb0R5ZTXrswtZtSuflTsLWLEjn+U78sgtrgCgfds2jOqdzLlH9SN9YBeO7N+F3vqmqIgETOVeo7Siio17ilifXci6rELWZhWyZlcBG/cUUVntALRv14bhaUmcNqYnY/qkMKZ3CiN7JeuiFyIScVpVueeVVLBtXzFbc4rZvLeYLTnFbNpbxKY9xezIK8FDHY4Z9OvSkeFpSZw0Ko0RPTsxslcyg7sn6vREEYkKMbsXLKMAAAVjSURBVFPuRWWVZBWUsTOvhN35pezKK2NHbgk780rYnlvKtn3FFJRWfuU5nTvGMbBbIhMHdWVgt0QGpyYytEcSg7onEh/XNqD/ExGRQxfV5f7PVVnc9dYKsvJLKSqv+trjKQlx9O6cQO+UeCYO7ELfLh3p0yWB/l070q9rR1IS4gJILSLS/MIqdzPrCswEvgPsAX7h7s/VM86A3wJX1GyaCfzcff+ER9Pq3DGOUb2S+dZhqfToFE+PTh3olRJPz5pbx/ZR/XeXiMhBC7f9HgbKgTRgHDDXzDLcPbPOuCuBs4CxgAP/ADYAjzZN3K8a378LD1/QpTleWkQkqjV6dNDMEoFpwO3uXujunwBvANPrGX4xcL+7b3P37cD9wCVNmFdERMIQzqkfw4Eqd19Ta1sGMLqesaNrHmtsnIiINKNwyj0JyKuzLQ/oFMbYPCDJ6vl6ppldaWYLzGxBdnZ2uHlFRCQM4ZR7IVD3isvJQEEYY5OBwvoOqLr74+6e7u7pqamp4eYVEZEwhFPua4B2Zjas1raxQN2DqdRsGxvGOBERaUaNlru7FwGvAneZWaKZHQOcCcyuZ/jTwE1m1sfMegP/CcxqwrwiIhKGcL9LfzWQAGQBzwMz3D3TzKaaWWGtcY8BbwLLgOXA3JptIiLSgsI6z93dcwidv153+zxCB1H333fgv2puIiISEGumL48eWAizbGDzQT69O6FvzUaaSM0FkZtNuQ6Mch2YWMw1wN3rPSMlIsr9UJjZAndPDzpHXZGaCyI3m3IdGOU6MK0tl9avFRGJQSp3EZEYFAvl/njQARoQqbkgcrMp14FRrgPTqnJF/Zy7iIh8XSzsuYuISB0qdxGRGKRyFxGJQTFX7mY2zMxKzeyZoLMAmNkzZrbTzPLNbI2ZXdH4s5o9Uwczm2lmm82swMwWm9lpQecCMLNra5aCLjOzWQFn6Wpmc8ysqOZndX6QeWoyRczPp7YIf09F3GewtubqrFi8yOjDwPygQ9RyL3C5u5eZ2QjgQzNb7O4LA8zUDtgKHA9sAU4HXjKzw919U4C5AHYA9wCnEFrPKEjhXl6yJUXSz6e2SH5PReJnsLZm6ayY2nM3s/OAXOD9oLPs5+6Z7l62/27NbUiAkXD3Ine/0903uXu1u78FbAQmBJmrJtur7v4asDfIHAd4eckWEyk/n7oi/D0VcZ/B/Zqzs2Km3M0sGbiL0DLDEcXM/mxmxcAqYCfwt4AjfYWZpRG6nKLW3v8/B3J5Sakj0t5TkfgZbO7OiplyB+4GZrr71qCD1OXuVxO6LOFUQmvjl33zM1qOmcUBzwJPufuqoPNEkAO5vKTUEonvqQj9DDZrZ0VFuZvZh2bmDdw+MbNxwEnAA5GUq/ZYd6+q+ad9X2BGJOQyszaELrpSDlzbnJkOJFeEOJDLS0qNln5PHYiW/Aw2piU6KyoOqLr7t77pcTO7ARgIbKm5FncS0NbMRrn7kUHlakA7mnm+L5xcNRctn0noYOHp7l7RnJnCzRVB/n15SXdfW7NNl438BkG8pw5Ss38Gw/AtmrmzomLPPQyPE/rDGldze5TQVaBOCTKUmfUws/PMLMnM2prZKcCPgA+CzFXjEWAkcIa7lwQdZj8za2dm8UBbQm/2eDNr8Z2QA7y8ZIuJlJ9PAyLuPRXBn8Hm7yx3j7kbcCfwTATkSAU+InQ0PJ/Q5Qd/HAG5BhA6Y6CU0PTD/tsFEZDtTv7vjIb9tzsDytIVeA0oInR63/n6+UTXeypSP4MN/Lk2aWdp4TARkRgUK9MyIiJSi8pdRCQGqdxFRGKQyl1EJAap3EVEYpDKXUQkBqncRURikMpdRCQG/X9A/lu2GB8+RgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_function(torch.sigmoid, title='Sigmoid', min=-4, max=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mnist_loss(predictions, targets):\n",
    "    predictions = predictions.sigmoid()\n",
    "    return torch.where(targets==1, 1-predictions, predictions).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SGD and Mini-Batches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[tensor([ 3, 12,  8, 10,  2]),\n",
       " tensor([ 9,  4,  7, 14,  5]),\n",
       " tensor([ 1, 13,  0,  6, 11])]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coll = range(15)\n",
    "dl = DataLoader(coll, batch_size=5, shuffle=True)\n",
    "list(dl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#26) [(0, 'a'),(1, 'b'),(2, 'c'),(3, 'd'),(4, 'e'),(5, 'f'),(6, 'g'),(7, 'h'),(8, 'i'),(9, 'j')...]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds = L(enumerate(string.ascii_lowercase))\n",
    "ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(tensor([17, 18, 10, 22,  8, 14]), ('r', 's', 'k', 'w', 'i', 'o')),\n",
       " (tensor([20, 15,  9, 13, 21, 12]), ('u', 'p', 'j', 'n', 'v', 'm')),\n",
       " (tensor([ 7, 25,  6,  5, 11, 23]), ('h', 'z', 'g', 'f', 'l', 'x')),\n",
       " (tensor([ 1,  3,  0, 24, 19, 16]), ('b', 'd', 'a', 'y', 't', 'q')),\n",
       " (tensor([2, 4]), ('c', 'e'))]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dl = DataLoader(ds, batch_size=6, shuffle=True)\n",
    "list(dl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting It All Together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = init_params((28*28,1))\n",
    "bias = init_params(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([256, 784]), torch.Size([256, 1]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dl = DataLoader(dset, batch_size=256)\n",
    "xb,yb = first(dl)\n",
    "xb.shape,yb.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_dl = DataLoader(valid_dset, batch_size=256)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4, 784])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch = train_x[:4]\n",
    "batch.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-11.1002],\n",
       "        [  5.9263],\n",
       "        [  9.9627],\n",
       "        [ -8.1484]], grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds = linear1(batch)\n",
    "preds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.5006, grad_fn=<MeanBackward0>)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss = mnist_loss(preds, train_y[:4])\n",
    "loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([784, 1]), tensor(-0.0001), tensor([-0.0008]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss.backward()\n",
    "weights.grad.shape,weights.grad.mean(),bias.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_grad(xb, yb, model):\n",
    "    preds = model(xb)\n",
    "    loss = mnist_loss(preds, yb)\n",
    "    loss.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(-0.0002), tensor([-0.0015]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calc_grad(batch, train_y[:4], linear1)\n",
    "weights.grad.mean(),bias.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(-0.0003), tensor([-0.0023]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calc_grad(batch, train_y[:4], linear1)\n",
    "weights.grad.mean(),bias.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights.grad.zero_()\n",
    "bias.grad.zero_();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_epoch(model, lr, params):\n",
    "    for xb,yb in dl:\n",
    "        calc_grad(xb, yb, model)\n",
    "        for p in params:\n",
    "            p.data -= p.grad*lr\n",
    "            p.grad.zero_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[False],\n",
       "        [ True],\n",
       "        [ True],\n",
       "        [False]])"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(preds>0.0).float() == train_y[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def batch_accuracy(xb, yb):\n",
    "    preds = xb.sigmoid()\n",
    "    correct = (preds>0.5) == yb\n",
    "    return correct.float().mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.5000)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_accuracy(linear1(batch), train_y[:4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def validate_epoch(model):\n",
    "    accs = [batch_accuracy(model(xb), yb) for xb,yb in valid_dl]\n",
    "    return round(torch.stack(accs).mean().item(), 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5219"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "validate_epoch(linear1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6883"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = 1.\n",
    "params = weights,bias\n",
    "train_epoch(linear1, lr, params)\n",
    "validate_epoch(linear1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8314 0.9017 0.9227 0.9349 0.9438 0.9501 0.9535 0.9564 0.9594 0.9618 0.9613 0.9638 0.9643 0.9652 0.9662 0.9677 0.9687 0.9691 0.9691 0.9696 "
     ]
    }
   ],
   "source": [
    "for i in range(20):\n",
    "    train_epoch(linear1, lr, params)\n",
    "    print(validate_epoch(linear1), end=' ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating an Optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_model = nn.Linear(28*28,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([1, 784]), torch.Size([1]))"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w,b = linear_model.parameters()\n",
    "w.shape,b.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BasicOptim:\n",
    "    def __init__(self,params,lr): self.params,self.lr = list(params),lr\n",
    "\n",
    "    def step(self, *args, **kwargs):\n",
    "        for p in self.params: p.data -= p.grad.data * self.lr\n",
    "\n",
    "    def zero_grad(self, *args, **kwargs):\n",
    "        for p in self.params: p.grad = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt = BasicOptim(linear_model.parameters(), lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_epoch(model):\n",
    "    for xb,yb in dl:\n",
    "        calc_grad(xb, yb, model)\n",
    "        opt.step()\n",
    "        opt.zero_grad()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4157"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "validate_epoch(linear_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(model, epochs):\n",
    "    for i in range(epochs):\n",
    "        train_epoch(model)\n",
    "        print(validate_epoch(model), end=' ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4932 0.8618 0.8203 0.9102 0.9331 0.9468 0.9555 0.9629 0.9658 0.9673 0.9687 0.9707 0.9726 0.9751 0.9761 0.9761 0.9775 0.978 0.9785 0.9785 "
     ]
    }
   ],
   "source": [
    "train_model(linear_model, 20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4932 0.852 0.8335 0.9116 0.9326 0.9473 0.9555 0.9624 0.9648 0.9668 0.9692 0.9712 0.9731 0.9746 0.9761 0.9765 0.9775 0.978 0.9785 0.9785 "
     ]
    }
   ],
   "source": [
    "linear_model = nn.Linear(28*28,1)\n",
    "opt = SGD(linear_model.parameters(), lr)\n",
    "train_model(linear_model, 20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dls = DataLoaders(dl, valid_dl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(dls, nn.Linear(28*28,1), opt_func=SGD,\n",
    "                loss_func=mnist_loss, metrics=batch_accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>batch_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.636857</td>\n",
       "      <td>0.503549</td>\n",
       "      <td>0.495584</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.545725</td>\n",
       "      <td>0.170281</td>\n",
       "      <td>0.866045</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.199223</td>\n",
       "      <td>0.184893</td>\n",
       "      <td>0.831207</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.086580</td>\n",
       "      <td>0.107836</td>\n",
       "      <td>0.911187</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.045185</td>\n",
       "      <td>0.078481</td>\n",
       "      <td>0.932777</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.029108</td>\n",
       "      <td>0.062792</td>\n",
       "      <td>0.946516</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.022560</td>\n",
       "      <td>0.053017</td>\n",
       "      <td>0.955348</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.019687</td>\n",
       "      <td>0.046500</td>\n",
       "      <td>0.962218</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.018252</td>\n",
       "      <td>0.041929</td>\n",
       "      <td>0.965162</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.017402</td>\n",
       "      <td>0.038573</td>\n",
       "      <td>0.967615</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit(10, lr=lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding a Nonlinearity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def simple_net(xb): \n",
    "    res = xb@w1 + b1\n",
    "    res = res.max(tensor(0.0))\n",
    "    res = res@w2 + b2\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w1 = init_params((28*28,30))\n",
    "b1 = init_params(30)\n",
    "w2 = init_params((30,1))\n",
    "b2 = init_params(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD7CAYAAABt0P8jAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXhU5fn/8fctW1hlC7iBCAUUkDXivlRtcZcWtbK4tFYURau2Vluh4lK3trZqKZZvtaggLgiudalVvu7aEDaDEJQdBMIWk7AEkvv7x0x+v3FMyAmZJTPzeV3XXMyc85xz7nOYuXPmOWfux9wdERHJHPslOwAREUksJX4RkQyjxC8ikmGU+EVEMowSv4hIhmmY7ACCaN++vXfp0iXZYYiIpJQ5c+Zscvfs6Okpkfi7dOlCbm5ussMQEUkpZrayqunq6hERyTBK/CIiGUaJX0Qkwyjxi4hkGCV+EZEMU2PiN7MmZvaYma00s2Izm2tmZ+6l/Y1mtt7MiszscTNrEjGvi5m9a2bbzWyxmZ0eqx0REZFggpzxNwRWAycD+wPjgefMrEt0QzMbAtwKnAZ0AboCd0Q0mQ7MBdoBtwEzzOw795iKiEj81Jj43b3U3Se4+wp3r3D3V4HlwKAqml8GPObu+e6+FbgLuBzAzHoAA4Hb3X2Hu78ALASGxWhfRETSxleFJfzxzSXsKa+I+bpr3cdvZh2BHkB+FbN7A/MjXs8HOppZu/C8Ze5eHDW/dzXbGW1muWaWW1hYWNswRURS1vayPYyZOoenP1vF5tKymK+/VonfzBoB04An3H1xFU1aAEURryuft6xiXuX8llVty90nu3uOu+dkZ6s3SEQyg7tz26zPWbqxhIcu7k/HVlkx30bgxG9m+wFPAWXA2GqalQCtIl5XPi+uYl7l/GJERASAaZ+uYtbctdx4eg9O7B6fk95Aid/MDHgM6AgMc/fd1TTNB/pFvO4HbHD3zeF5Xc2sZdT8qrqMREQyzoI127jzlUWc0jObsd//Xty2E/SMfxJwBHCuu+/YS7sngSvMrJeZtQHGAVMA3L0AmAfcbmZZZvYjoC/wwr4GLyKSLraWljFmah7ZLZvw54v6s99+FrdtBbmP/1DgKqA/sN7MSsKPkWbWOfy8M4C7vwE8ALwLrAw/bo9Y3cVADrAVuA+4wN115VZEMlpFhXPjc/MoLN7F30YOpE3zxnHdXo1lmd19JbC3Pz0toto/CDxYzbpWAKcED09EJP1NfPdLZi8p5K6hfejXqXXct6eSDSIiSfTB0k08+HYBQ/sfxKijOydkm0r8IiJJ8nXRDq5/Zi7dO7Tgnh8fSeg+mvhT4hcRSYKyPRVcMy2PXbvLmTRqEM0aJ25AxJQYelFEJN3c868vmLtqGxNHDKRbdouaF4ghnfGLiCTYqwvWMeWjFfz0+C6c3ffAhG9fiV9EJIG+3FjCLTMWMLBza35z5hFJiUGJX0QkQUp3hYqvNWnUgIkjB9K4YXJSsPr4RUQSwN357ayFfFlYwlM/O5oD92+atFh0xi8ikgBTP1nJS/PWcdPpPTihe/ukxqLELyISZ/NWb+POVxfx/Z7ZXBvH4mtBKfGLiMTR1tIyrp2WR4eWWfz5J/EtvhaU+vhFROKkosK54dlQ8bUZY46ldbP4Fl8LSmf8IiJx8sg7X/K/BYXcfl4v+h4S/+JrQSnxi4jEwXsFhfzlPwX8eMDBjBicmOJrQSnxi4jE2NptO/jFM3Pp0aElv/9R4oqvBRV06MWxZpZrZrvMbMpe2j0aMVBLSbh9ccT82Wa2M2L+khjsg4hIvVG2p4Jrp+Wxu9yZNGogTRs3SHZI3xH04u464G5gCFDtrw7c/Wrg6srX4T8SFVHNxrr7P2oXpohIavj9a4uYt3obfxs5kK4JLr4WVKDE7+4zAcwsBzgkyDJm1hwYBpyzz9GJiKSQl+ev44mPV/LzEw7jrCMTX3wtqHj28Q8DCoH3oqbfa2abzOxDMzuluoXNbHS4eym3sFDD8opI/bZ0QzG3vrCAnEPbcMuZhyc7nL2KZ+K/DHjS3T1i2i1AV+BgYDLwipl1q2phd5/s7jnunpOdnR3HMEVE6qZ01x7GTMujWeNQ8bVGDer3fTNxic7MOgEnA09GTnf3T9292N13ufsTwIfAWfGIQUQkEdydW2cuZFlhCQ8PH0DHVlnJDqlG8fqzdCnwkbsvq6GdA/XrPicRkVp48uOVvDJ/Hb/8YU+O65bc4mtBBb2ds6GZZQENgAZmlmVme7swfCkwJWodrc1sSOWyZjYSOAl4cx9jFxFJqrxVW7n7tUWcengHxpxcZa91vRT0jH8csAO4FRgVfj7OzDqH78f/fz9LM7NjCd3583zUOhoRuiW0ENgEXAcMdXfdyy8iKWdLaRljp+XRsVUWf76ofhRfCyro7ZwTgAnVzP7Wjaru/jHQvIp1FAJH1S48EZH6p7zC+cUzc9lUWsbMMcexf7NGyQ6pVur3pWcRkXro4f8s5f2lm7jjvN70OXj/ZIdTa0r8IiK1MHvJRh5+ZynDBh7CxUd1SnY4+0SJX0QkoDVbt3PDs/Po2bEldw/tU++KrwWlxC8iEsCuPeVcOy2P8nJn0qhB9bL4WlAagUtEJIC7X/2C+WuKeHTUIA5r/537V1KKzvhFRGrw0ry1PPXJSkaf1JUz+hyQ7HDqTIlfRGQvCjYUc+sLCxncpS03D+mZ7HBiQolfRKQaJbv2cPXUOTRv0pBHRgyo98XXgkqPvRARiTF355YXFrBiUykPD++fEsXXglLiFxGpwpSPVvDagq+5ecjhKVN8LSglfhGRKHNWbuX3r33B6Ud05OqTuyY7nJhT4hcRibC5ZBdjn87joNZN+dNF/VL2R1p7o/v4RUTCQsXX5rG5svha09QqvhaUzvhFRMIeeruAD77cxF3np2bxtaCU+EVEgHcXb+Thd77kwkGH8JOjOte8QAoLOgLXWDPLNbNdZjZlL+0uN7Py8OAslY9TIuZ3MbN3zWy7mS02s9PrvgsiInWzekuo+NoRB7birqF9kh1O3AXt419HaPSsIUDTGtp+7O4nVDNvOvAxoQHWzwJmmFn38CAtIiIJt2tPOdc+nUdFhTNp5ECyGqVu8bWgAp3xu/tMd38R2LyvGzKzHsBA4HZ33+HuLwALgWH7uk4Rkbq685VFLFhTxB8v6keXFC++FlQ8+vgHmNkmMysws/ERg7L3Bpa5e3FE2/nh6d9hZqPD3Uu5hYX6QiAisTczbw3TPl3FVSd1ZUjv1C++FlSsE/97QB+gA6Ez+eHAzeF5LYCiqPZFQMuqVuTuk909x91zsrOzYxymiGS6xeu/4bezFjL4sPQpvhZUTBO/uy9z9+XuXuHuC4E7gQvCs0uAVlGLtAKKERFJoOKduxkzNY+WWY3464gBNEyT4mtBxXtvHaj82Vs+0NXMIs/w+4Wni4gkhLvz6xkLWLVlO38dPoAOLdOn+FpQQW/nbGhmWUADoIGZZUX03Ue2O9PMOoafHw6MB14CcPcCYB5we3j5HwF9gRdisysiIjV77IPlvP75en49pCdHd22X7HCSIugZ/zhgB3ArMCr8fJyZdQ7fq1/5a4fTgAVmVgr8C5gJ3BOxnouBHGArcB9wgW7lFJFEyV2xhfteX8wPe3Vk9EnpV3wtKHP3ZMdQo5ycHM/NzU12GCKSwjaV7OLsh98nq1EDXh57QtrW4YlkZnPcPSd6uoq0iUjaK69wrp8+l23bdzPrmsEZkfT3RolfRNLeg/9ewkdfbeaBC/rS66DomwszT2bdwyQiGeedxRuY+O5X/CSnExfldEp2OPWCEr+IpK3VW7ZzwzPz6HVgK+44v8oiARlJiV9E0tLO3eVcMy0PBx4dNSgjiq8FpT5+EUlLd7yyiIVri/ifS3Po3K5ZssOpV3TGLyJp54U5a5j+2SquPrkbP+jVMdnh1DtK/CKSVhav/4bbXlzIMV3b8qsf9kh2OPWSEr+IpI3K4mutshrxyPCBGVd8LSj18YtIWnB3bn4+VHxt+pXHkN2ySbJDqrf051BE0sJjHyznjfz13HrG4Qw+rG2yw6nXlPhFJOV9tnwL976+mDN6H8DPTzws2eHUe0r8IpLSNhbvZOzTeXRq05QHLuyLmdW8UIZTH7+IpKw95RVcP30u3+zczRM/G0yrrMwuvhZU0IFYxoYHPt9lZlP20u4yM5tjZt+Y2RozeyBywBYzm21mO8M1/EvMbEkM9kFEMtSf/l3AJ8u2cPfQIzniQBVfCypoV8864G7g8RraNQNuANoDRxMamOVXUW3GunuL8COzRjgWkZj596INTJr9FcMHd+KCQYckO5yUEqirx91nAphZDlDtEXb3SREv15rZNOD7dYpQRCTKqs3buem5efQ5uBW3n6via7UV74u7J/HdwdTvNbNNZvahmZ1S3YJmNjrcvZRbWKjRGUUkZOfucsZMm4MBk0aq+Nq+iFviN7OfEhpf948Rk28BugIHA5OBV8ysW1XLu/tkd89x95zs7Ox4hSkiKWbCy/nkr/uGv1zcn05tVXxtX8Ql8ZvZUEKDqZ/p7psqp7v7p+5e7O673P0J4EPgrHjEICLp57nc1Tzz39Vc+/1unHq4iq/tq5jfzmlmZwD/A5zt7gtraO6AbroVkRrlryti/Iufc1y3dtz0A90XUhdBb+dsaGZZQAOggZllRd6mGdHuVGAaMMzdP4ua19rMhlQua2YjCV0DeLPuuyEi6axox26umZZH62aNeHj4ABrsp/PFugja1TMO2AHcCowKPx9nZp3D9+N3DrcbD+wP/CviXv3Xw/MaEboltBDYBFwHDHV33csvItUKFV+bz9qtO5g4YiDtW6j4Wl0FvZ1zAjChmtktItpVe+umuxcCR9UiNhERJr+3jLcWbWDc2UeQ00XF12JBtXpEpN76dNlmHnhzCWcdeQBXnKDia7GixC8i9dLGb3YydvpcDm3bjPuHqfhaLKlIm4jUO3vKK7hu+lxKdu5h6hVH01LF12JKiV9E6p0/vLWET5dv4c8/6UfPA1omO5y0o64eEalX3spfz9//dxkjj+7Mjwao+Fo8KPGLSL2xcnMpv3x+Pn0P2Z/fndsr2eGkLSV+EakXdu4uZ8zUPPYzY+KIgTRpqOJr8aI+fhGpF3730ucs+vob/nn5USq+Fmc64xeRpHv2v6t4LncN1536Pb5/eIdkh5P2lPhFJKk+X1vE+JfyOeF77bnh9B7JDicjKPGLSNJUFl9r26wxD13cX8XXEkR9/CKSFBUVzi+fm8+6bTt49qpjaKfiawmjM34RSYq/v7eMt7/YwG/POoJBh6r4WiIp8YtIwn381Wb+8OZizu57ID89vkuyw8k4SvwiklAbv9nJddPn0qV9cxVfS5KgI3CNNbNcM9tlZlNqaHujma03syIze9zMmkTM62Jm75rZdjNbbGan1zF+EUkhu8srGPv0XEp37eHRUYNo0USXGZMh6Bn/OkKjZz2+t0ZmNoTQKF2nAV2ArsAdEU2mA3OBdsBtwAwzy65dyCKSqv7w5hI+W7GF+4YdSY+OKr6WLIESv7vPdPcXgc01NL0MeMzd8919K3AXcDmAmfUABgK3u/sOd38BWAgM29fgRSR1vPH5eia/t4xLjjmU8/sfnOxwMlqs+/h7A/MjXs8HOppZu/C8Ze5eHDW/d1UrMrPR4e6l3MLCwhiHKSKJtGJTKTc/P59+nVoz7pwjkh1Oxot14m8BFEW8rnzesop5lfOr/L7n7pPdPcfdc7Kz1Rskkqp2lJVz9dQ5NGhgTBwxQMXX6oFYX1kpAVpFvK58XlzFvMr5xYhIWnJ3xr/0OUs2FPPPy4/ikDYqvlYfxPqMPx/oF/G6H7DB3TeH53U1s5ZR8/NjHIOI1BPP/nc1M+as4bpTu3NKTxVfqy+C3s7Z0MyygAZAAzPLMrOqvi08CVxhZr3MrA0wDpgC4O4FwDzg9vDyPwL6Ai/EYD9EpJ75fG0Rv3s5nxO7t+cXp3VPdjgSIegZ/zhgB6FbNUeFn48zs85mVmJmnQHc/Q3gAeBdYGX4cXvEei4GcoCtwH3ABe6uK7ciaaZo+26unjqHds0b89DFA1R8rZ4xd092DDXKycnx3NzcZIchIgFUVDhXPpnLe0sLee6qYxnQuU2yQ8pYZjbH3XOip6tkg4jE1KPvfcV/Fm9k3Nm9lPTrKSV+EYmZj77axB/fXMK5/Q7i0mMPTXY4Ug0lfhGJifVFO7l++lwOa9+c+358pIqv1WOqkCQidRYqvpbH9rJypl95DM1VfK1e0/+OiNTZ/a8vJnflVh4ePoDuKr5W76mrR0Tq5PWFX/OPD5Zz2bGHcl6/g5IdjgSgxC8i+2xZYQk3z1hA/06tue3sXskORwJS4heRfbKjrJxrpuXRqIExceRAGjdUOkkV6uMXkVpzd257cSFLNhTzxE8Hc3DrpskOSWpBf6JFpNamf7aamXlr+cVp3Tmph8qmpxolfhGplQVrtjHh5XxO6pHN9aeq+FoqUuIXkcC2bS9jzNQ82rdozF9+0p/9VHwtJamPX0QCqahwbnx2HhuLd/L81cfRtnnjZIck+0hn/CISyN9mf8m7SwoZf04v+ndqnexwpA6U+EWkRh9+uYkH/13Aef0O4pJjVHwt1QUdgautmc0ys1IzW2lmI6pp93p4YJbKR5mZLYyYv8LMdkTMfytWOyIi8VFZfK1rdgvuVfG1tBC0j38iUAZ0BPoDr5nZfHf/1ni57n5m5Gszmw28E7Wuc9397X0LV0QSaXd5Bdc+ncfO3eU8OmqQiq+liRrP+M2sOTAMGO/uJe7+AfAycEkNy3UBTgSeqnuYIpIM9/5rMXNWbuX+C/ryvQ4tkh2OxEiQrp4eQHl4sPRK84HeNSx3KfC+uy+Pmj7NzArN7C0z61fdwmY22sxyzSy3sFDD8ook2msLvubxD5dz+XFdOKeviq+lkyCJvwVQFDWtCKip9uqlwJSoaSOBLsChhAZkf9PMqrw9wN0nu3uOu+dkZ+uXgSKJ9FVhCb+eMZ+BnVvz27OOSHY4EmNBEn8J0CpqWiuguLoFzOwE4ABgRuR0d//Q3Xe4+3Z3vxfYRqg7SETqie1lexgzdQ5NGjVQ8bU0FeR/tABoaGaRv83uB+RX0x7gMmCmu5fUsG4HdIuASD3h7tw263OWbizhoYv7c+D+Kr6WjmpM/O5eCswE7jSz5mZ2PHA+1Vy0NbOmwIVEdfOYWWczO97MGptZlpndDLQHPqzjPohIjEz7dBWz5q7lxtN7cGJ3dbGmq6Df4a4BmgIbgenAGHfPN7MTzSz6rH4ooWsA70ZNbwlMArYCa4EzgDPdffO+Bi8isTN/9TbufGURp/TMZuz3v5fscCSOzN2THUONcnJyPDc3N9lhiKStraVlnPPIBwC8et0JtFEdnrRgZnPcPSd6un6NIZLhKiqcG5+bR2HxLp6/+lgl/Qygy/UiGe6v737J7CWFjD+3F/1UfC0jKPGLZLD3lxby57cLGNr/IEYd3TnZ4UiCKPGLZKh123bwi2fm0b1DC+5R8bWMosQvkoHK9lQw9uk8du0uZ9KoQTRrrMt9mUT/2yIZ6J5/fUHeqm1MHDGQbtkqvpZpdMYvkmFemb+OKR+t4GfHH8bZfQ9MdjiSBEr8Ihnky40l3PrCAgYd2obfnHV4ssORJFHiF8kQpbtCxdeyGjVg4oiBNGqgj3+mUh+/SAZwd347ayFfFZbw1BVHc8D+WckOSZJIf/JFMsDUT1by0rx13PSDHhz/vfbJDkeSTIlfJM3NXbWVO19dxKmHd+CaU1R8TZT4RdLaltIyrp2WR8dWWTx4UT/2208/0hL18YukrfIK54Zn57GppIwZY46ldTMVX5OQQGf8ZtbWzGaZWamZrTSzEdW0m2Bmu82sJOLRNWJ+fzObY2bbw//2j9WOiMi3PfLOUt4rKOT283rR9xAVX5P/L2hXz0SgDOhIaMD0SWbWu5q2z7p7i4jHMgAzawy8BEwF2gBPAC+Fp4tIDP1vQSEP/WcpPx5wMCMGq/iafFuNid/MmgPDgPHuXuLuHwAvA5fUclunEOpa+ou773L3hwmNt3tqLdcjInuxdtsObnhmLj06tOT3P1LxNfmuIGf8PYBydy+ImDYfqO6M/1wz22Jm+WY2JmJ6b2CBf3vIrwXVrcfMRptZrpnlFhYWBghTRHbtKeeaaXnsLncmjRpI08YNkh2S1ENBEn8LQmPoRioiNIZutOeAI4Bs4Ergd2Y2fB/Wg7tPdvccd8/JztagzyJB/P61L5i/eht/vLAvXVV8TaoRJPGXAK2iprUCiqMbuvsid1/n7uXu/hHwEHBBbdcjIrX30ry1PPnxSn5+wmGc0UfF16R6QRJ/AdDQzLpHTOsH5AdY1gn14xNu39e+3eHYN+B6RGQvlm4o5jczF3JUlzbccqaKr8ne1Zj43b0UmAncaWbNzex44Hzgqei2Zna+mbWxkMHA9YTu5AGYDZQD15tZEzMbG57+Tgz2QyRjlezaw9VT59CscQMeGa7ia1KzoO+Qa4CmwEZgOjDG3fPN7EQzK4lodzHwJaHumyeB+939CQB3LwOGApcC24CfAUPD00VkH7g7t76wgOWbSnl4+AAVX5NAAv1y1923EEra0dPfJ3TRtvL18Og2Ue3nAoNqGaOIVOOJj1bw6oKvuXlIT47rpuJrEoy+E4qkqDkrt3L3a19w2uEdGHNyt2SHIylEiV8kBW0u2cXYp/M4sHUWD17UX8XXpFZUpE0kxVQWX9tcWsbMMcexf7NGyQ5JUozO+EVSzEP/Wcr7Szdxx3m96XPw/skOR1KQEr9ICpm9ZCOPvLOUYQMP4eKjOiU7HElRSvwiKWLN1u3c8Ow8enZsyd1D+6j4muwzJX6RFFBZfK283Jk0apCKr0md6OKuSAq469VFLFhTxKOjBnFY++bJDkdSnM74Req5F+euZeonqxh9UlfO6HNAssORNKDEL1KPFYSLrw3u0pabh/RMdjiSJpT4ReqpyuJrzZs05K8jBqj4msSM3kki9ZC7c8uMBazYVMojwwfQoZWKr0nsKPGL1EP//HAFry38mpuHHM6x3dolOxxJM0r8IvVM7oot3POvLzj9iI5cdVLXZIcjaUiJX6Qe2VSyi2ufzuOg1k3500X9VHxN4iJQ4jeztmY2y8xKzWylmY2opt3NZva5mRWb2XIzuzlq/goz22FmJeHHW7HYCZF0UF7h/OKZuWzdvpu/jRzI/k1VfE3iI+gPuCYCZUBHoD/wmpnNd/fo8XKN0AhbC4BuwFtmttrdn4loc667v13HuEXSzp//XcCHX27m/mFHqviaxFWNZ/xm1hwYBox39xJ3/wB4Gbgkuq27P+Duee6+x92XEBpv9/hYBy2Sbt5ZvIG/vvslFw46hJ8c1TnZ4UiaC9LV0wMod/eCiGnzgd57W8hCFaROBKK/FUwzs0Ize8vM+u1l+dFmlmtmuYWFhQHCFElNq7ds58Zn53PEga24a2ifZIcjGSBI4m8BFEVNKwJa1rDchPD6/xkxbSTQBTgUeBd408xaV7Wwu0929xx3z8nOzg4Qpkjq2bk7VHytwp1HRw0kq5GKr0n8BUn8JUCrqGmtgOLqFjCzsYT6+s92912V0939Q3ff4e7b3f1eYBuhbwUiGenOVxexcG0Rf7qwH4e2U/E1SYwgib8AaGhm3SOm9eO7XTgAmNnPgFuB09x9TQ3rdkIXhEUyzsy8NTz96SquPrkbP+yt4muSODUmfncvBWYCd5pZczM7HjgfeCq6rZmNBO4BfuDuy6LmdTaz482ssZllhW/1bA98GIsdEUkli9d/w29nLeSYrm351Q97JDscyTBBf8B1DdAU2AhMB8a4e76ZnWhmJRHt7gbaAf+NuFf/0fC8lsAkYCuwFjgDONPdN8diR0RSRfHO3YyZmkerrEY8PHwADVV8TRIs0H387r4FGFrF9PcJXfytfH3YXtaRD/TdhxhF0oa78+sZC1i1ZTvTrzyGDi1VfE0ST6caIgn02AfLef3z9dxyRk8GH9Y22eFIhlLiF0mQ/67Ywr2vL2ZI745ceaKKr0nyKPGLJEBh8S6unZZHpzZN+cOF/Qj9vlEkOTTYukic7Smv4PrpcynasZspPx1MqywVX5PkUuIXibMH/13Ax8s284cL+tLroOjfQooknrp6ROLo7UUb+Nvsr7j4qE5cmNMp2eGIAEr8InGzavN2bnpuHr0PasWE8/Za01AkoZT4ReJg5+5yrnl6DgCTRg5S8TWpV9THLxIHd7ySz+drv+Efl+bQuV2zZIcj8i064xeJsRlz1jD9s9Vcc0o3Tu/VMdnhiHyHEr9IDH3x9TfcNmshx3Ztx00/UPE1qZ+U+EVi5JuduxkzdQ77N1XxNanf1McvEgPuzq+fX8DqrTt4ZvQxZLdskuyQRKqlUxKRGPjH+8t5I389vznzcI7qouJrUr8p8YvU0afLNnPfG4s5s88BXHFCtZXJReqNQInfzNqa2SwzKzWzlWY2opp2Zmb3m9nm8OMBi6hGZWb9zWyOmW0P/9s/VjsikgyfLNvMtU/PpXPbZjxwQV8VX5OUEPSMfyJQBnQERgKTzKyqnyKOJjRgSz9Cg66cA1wFYGaNgZeAqUAb4AngpfB0kZRSvHM3t81ayMWTP6FZ4wb8/ZJBtFTxNUkRNV7cNbPmwDCgj7uXAB+Y2cvAJYQGVY90GfCnykHWzexPwJXAo8Ap4e39xd0deNjMfgWcCrwRm935tp8/8V9Wbt4ej1VLhttUsouiHbv5+QmH8csf9qRpY/0yV1JHkLt6egDl7l4QMW0+cHIVbXuH50W26x0xb0E46VdaEJ7+ncRvZqMJfYOgc+fOAcL8rs5tm9O4oS5jSOz1PqgVlx3XhQGd2yQ7FJFaC5L4WwBFUdOKCA2eXlPbIqBFuJ+/NuvB3ScDkwFycnK8qjY1+d25vfZlMRGRtBbkdLgEiC4i3gooDtC2FVASPsuvzXpERCROgiT+AqChmXWPmNYPyK+ibX54XlXt8oG+9u3bHvpWsx4REYmTGhO/u5cCM4E7zay5mR0PnA88VUXzJ4GbzOxgMzsI+CUwJTxvNlAOXG9mTcxsbHj6O9VBRU4AAAUOSURBVHXbBRERqY2gVz6vAZoCG4HpwBh3zzezE82sJKLd34FXgIXA58Br4Wm4exmhWz0vBbYBPwOGhqeLiEiC2LdvsqmfcnJyPDc3N9lhiIikFDOb4+450dN1r6OISIZR4hcRyTBK/CIiGSYl+vjNrBBYuY+Ltwc2xTCcWFFctaO4akdx1U66xnWou2dHT0yJxF8XZpZb1cWNZFNctaO4akdx1U6mxaWuHhGRDKPELyKSYTIh8U9OdgDVUFy1o7hqR3HVTkbFlfZ9/CIi8m2ZcMYvIiIRlPhFRDKMEr+ISIZJq8QfLvf8mJmtNLNiM5trZmfWsMyNZrbezIrM7HEzaxKn2MaaWa6Z7TKzKTW0vdzMys2sJOJxSrLjCrdP1PFqa2azzKw0/P85Yi9tJ5jZ7qjj1TWRcVjI/Wa2Ofx4IGrsiZirRWxxOz5VbKs27/OEvJdqE1ciP3vh7dUqZ8XqmKVV4ic0lORqQuMB7w+MB54zsy5VNTazIYQGjD8N6AJ0Be6IU2zrgLuBxwO2/9jdW0Q8Zic7rgQfr4lAGdARGAlMMrPee2n/bNTxWpbgOEYTKjvej9AAQ+cAV8UohrrGBvE7PtECvZ8S/F4KHFdYoj57UIucFdNj5u5p/SA0oPuwauY9DdwT8fo0YH2c47kbmFJDm8uBDxJ8nILElZDjBTQnlNB6REx7CrivmvYTgKnJjAP4CBgd8foK4JM4/n/VJra4HJ+6vJ+S8dkLGFfCP3tVxFBlzorlMUu3M/5vMbOOQA+qH96xNzA/4vV8oKOZtYt3bAEMMLNNZlZgZuPNrGGyAyJxx6sHUO7uBVHb2tsZ/7lmtsXM8s1sTBLiqOrY7C3eRMYG8Tk+daHPXhVqyFkxO2Zpm/jNrBEwDXjC3RdX06wFUBTxuvJ5y3jGFsB7QB+gAzAMGA7cnNSIQhJ1vKK3U7mt6rbzHHAEkA1cCfzOzIYnOI6qjk2LOPbz1ya2eB2futBnL0qAnBWzY5ZSid/MZpuZV/P4IKLdfoS+9pYBY6tdIZQArSJeVz4vjkdcQbn7Mndf7u4V7r4QuBO4oLbriXVcJO54RW+ncltVbsfdF7n7Oncvd/ePgIfYh+NVhdrEUdWxKfHwd/I4CBxbHI9PXcTkvRRrsfrs1VbAnBWzY5ZSid/dT3F3q+ZxAoTurgAeI3TBa5i7797LKvMJXYyr1A/Y4O6bYx1XHTlQ6zPHOMSVqONVADQ0s+5R26quy+47m2AfjlcVahNHVccmaLzxji1arI5PXcTkvZQAcT9WtchZMTtmKZX4A5pE6Gvtue6+o4a2TwJXmFkvM2sDjAOmxCMoM2toZllAA6CBmWVV13doZmeG+/ows8MJXel/KdlxkaDj5e6lwEzgTjNrbmbHA+cTOiOqah/ON7M2FjIYuJ4YHK9axvEkcJOZHWxmBwG/JE7vpdrGFq/jU5VavJ8S9tmrTVyJ/OxFCJqzYnfMknn1OtYP4FBCf6F3EvpaVPkYGZ7fOfy6c8QyNwEbgG+AfwJN4hTbhHBskY8JVcUF/DEcUymwjNDXzUbJjivBx6st8GL4GKwCRkTMO5FQN0rl6+nA5nCsi4Hr4x1HFTEY8ACwJfx4gHAtrDi+34PGFrfjE/T9lMz3Um3iSuRnL7y9anNWPI+ZirSJiGSYdOzqERGRvVDiFxHJMEr8IiIZRolfRCTDKPGLiGQYJX4RkQyjxC8ikmGU+EVEMsz/AXPhmYaLqaN6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_function(F.relu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "simple_net = nn.Sequential(\n",
    "    nn.Linear(28*28,30),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(30,1)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(dls, simple_net, opt_func=SGD,\n",
    "                loss_func=mnist_loss, metrics=batch_accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>batch_accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.305828</td>\n",
       "      <td>0.399663</td>\n",
       "      <td>0.508341</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.142960</td>\n",
       "      <td>0.225702</td>\n",
       "      <td>0.807655</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.079516</td>\n",
       "      <td>0.113519</td>\n",
       "      <td>0.919529</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.052391</td>\n",
       "      <td>0.076792</td>\n",
       "      <td>0.943081</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.039796</td>\n",
       "      <td>0.060083</td>\n",
       "      <td>0.956330</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.033368</td>\n",
       "      <td>0.050713</td>\n",
       "      <td>0.963690</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.029680</td>\n",
       "      <td>0.044797</td>\n",
       "      <td>0.965653</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.027290</td>\n",
       "      <td>0.040729</td>\n",
       "      <td>0.968106</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.025568</td>\n",
       "      <td>0.037771</td>\n",
       "      <td>0.968597</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.024233</td>\n",
       "      <td>0.035508</td>\n",
       "      <td>0.970559</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.023149</td>\n",
       "      <td>0.033714</td>\n",
       "      <td>0.972031</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.022242</td>\n",
       "      <td>0.032243</td>\n",
       "      <td>0.972522</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.021468</td>\n",
       "      <td>0.031006</td>\n",
       "      <td>0.973503</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.020796</td>\n",
       "      <td>0.029944</td>\n",
       "      <td>0.974485</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.020207</td>\n",
       "      <td>0.029016</td>\n",
       "      <td>0.975466</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>0.019683</td>\n",
       "      <td>0.028196</td>\n",
       "      <td>0.976448</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>0.019215</td>\n",
       "      <td>0.027463</td>\n",
       "      <td>0.976448</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>0.018791</td>\n",
       "      <td>0.026806</td>\n",
       "      <td>0.976938</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>0.018405</td>\n",
       "      <td>0.026212</td>\n",
       "      <td>0.977920</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>0.018051</td>\n",
       "      <td>0.025671</td>\n",
       "      <td>0.977920</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>0.017725</td>\n",
       "      <td>0.025179</td>\n",
       "      <td>0.977920</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21</td>\n",
       "      <td>0.017422</td>\n",
       "      <td>0.024728</td>\n",
       "      <td>0.978410</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>22</td>\n",
       "      <td>0.017141</td>\n",
       "      <td>0.024313</td>\n",
       "      <td>0.978901</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>23</td>\n",
       "      <td>0.016878</td>\n",
       "      <td>0.023932</td>\n",
       "      <td>0.979392</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>24</td>\n",
       "      <td>0.016632</td>\n",
       "      <td>0.023580</td>\n",
       "      <td>0.979882</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>25</td>\n",
       "      <td>0.016400</td>\n",
       "      <td>0.023254</td>\n",
       "      <td>0.979882</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>26</td>\n",
       "      <td>0.016181</td>\n",
       "      <td>0.022952</td>\n",
       "      <td>0.979882</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27</td>\n",
       "      <td>0.015975</td>\n",
       "      <td>0.022672</td>\n",
       "      <td>0.980864</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>28</td>\n",
       "      <td>0.015779</td>\n",
       "      <td>0.022411</td>\n",
       "      <td>0.980864</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>29</td>\n",
       "      <td>0.015593</td>\n",
       "      <td>0.022168</td>\n",
       "      <td>0.981845</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30</td>\n",
       "      <td>0.015417</td>\n",
       "      <td>0.021941</td>\n",
       "      <td>0.981845</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>31</td>\n",
       "      <td>0.015249</td>\n",
       "      <td>0.021728</td>\n",
       "      <td>0.981845</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32</td>\n",
       "      <td>0.015088</td>\n",
       "      <td>0.021529</td>\n",
       "      <td>0.981845</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>33</td>\n",
       "      <td>0.014935</td>\n",
       "      <td>0.021341</td>\n",
       "      <td>0.981845</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>34</td>\n",
       "      <td>0.014788</td>\n",
       "      <td>0.021164</td>\n",
       "      <td>0.981845</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>35</td>\n",
       "      <td>0.014647</td>\n",
       "      <td>0.020998</td>\n",
       "      <td>0.982336</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>36</td>\n",
       "      <td>0.014512</td>\n",
       "      <td>0.020840</td>\n",
       "      <td>0.982826</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>37</td>\n",
       "      <td>0.014382</td>\n",
       "      <td>0.020691</td>\n",
       "      <td>0.982826</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>38</td>\n",
       "      <td>0.014257</td>\n",
       "      <td>0.020550</td>\n",
       "      <td>0.982826</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>39</td>\n",
       "      <td>0.014136</td>\n",
       "      <td>0.020415</td>\n",
       "      <td>0.982826</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit(40, 0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD+CAYAAADBCEVaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAZfElEQVR4nO3df5Dc9X3f8efrfkv3AxA6JLABBZBsLCei9TlxQ3CSxjGxmwwE7GkMxnRSBxfGE6ZOm3gypqW4M9RuO5k2Q7CZIcbGrmwngQSHGjvT4BjsukbEFa5sfOfElnAMdycJ7m5Pd7e3d+/+8d09rVZ7d1+dVtrd7/f1mLnR7Xe/t/fmo9OLz32+n31/FRGYmVm2dDS7ADMzazyHu5lZBjnczcwyyOFuZpZBDnczswxyuJuZZZDD3cwsg1KFu6T3S9onaUHSQ+uc+68lvSRpStIfS+ptSKVmZpaa0ryJSdINwDJwLbApIv7FKuddC3wK+KfAj4FHgW9ExAfXev2tW7fGjh07TqlwM7O8e/bZZw9HxHC957rSvEBEPAIgaQR49Rqn3go8GBEHyud/GPgMsGa479ixg3379qUpxczMyiQdXO25Rq+57wb2Vz3eD2yTdH6Dv4+Zma2h0eE+AExVPa58Plh7oqTbyuv4+yYnJxtchplZvjU63AvAUNXjyucztSdGxAMRMRIRI8PDdZeMzMxsgxod7geAPVWP9wDjEXGkwd/HzMzWkHYrZJekPqAT6JTUJ6nexdhPAf9S0usknQd8CHioYdWamVkqaWfuHwLmSHa9vLv8+YckXSKpIOkSgIh4Avgo8CRwsPzx7xtetZmZrSnVPvczbWRkJLwV0szs1Eh6NiJG6j2Xap+7mVk9EcFCaZnCQonl5eZPFOsJYHFpmWJpmcWloFhapri0RLEUFFeOJ38WS8ssLC2zWFpeea60tHxG6xvZsYU372r8phKHu1mTlZaSIFksBQtLSyshUx1EC+XHp/Ob9tJyUFgoMbuwxOxCiZmFErPlj5mFEsXS6iEWAQul5Osqr1Eof77UoqHeSNKZe+1/9fOXO9zN1lIJybni0gnhUwmkyuel0wij5QgWS0FxaWklfBfKYVysmfEVq2aDi0vLJ8wSF6qONzMb+3s66e/tYqC3i56uDrRGivV2dTDQ28XwYC8Dvd0M9HYy0Ne18vWdHWcwAU9Td0cHPV0ddHcmfyafi97qY1XP9XZ20t0lejo76Opsz/6KDndruNLSclWYLlFYWKRQni0W5k8O3NrHx4pLye/Sq1iOWAnPZoVkZ4fo7lQ5EDrprQqMSlB0d3Yw2Nd1QoBUB0nvSWFTCZaTgyj5PlozfNfTITFQDuL+3k76e7roaOFAttPjcM+ppeWoWmNcOnEZoLTMfGlpJXRnF0rMzJd/nS9WPq8K5/kSs8Xj5y2s8et9tcpMcKCvi/6eJHQuGOxjU08nHWuEmOCE0KsNyb6uDgb6kpllZVaZBFry0XOaM7Gero6WnqWagcM9MyKCV44t8tL0PC9NzzMxPc+R2SIvzxY5OrvIy8eKHJ0trvw5M1/a0PfZ1F0JzM6VUL7wnL4Tfj3v70kCe7AcpgN9XXWDtrtNf901awcO9xZTWCjxwtFjHDp6jJdniyet31Zf1T8yW2RieoGXpucZn56vO2Pu6+7g/P5ezuvv5rzNPVx6/mbO29zDuZu76evuTJYP6iwF9HZ10N/bxeAJod3ZtuuPZnnjcD+LqmfX49PzvDg1z6Gjx3ih8vHyHEdni2u+RmeHyuu54rz+HrYN9XHVxeey/Zw+tg31sW2ol+1DyedbB3rZ1NN5lv7rzKyVONzPgIjgbw+9wpcPvMSPp+YZn5pfdXbd1SFedd4mLtmymWsvOodLtmzm4i3J4/MHeleu4FfWlb3Wa2ZpONwb6OXZIo986x/43DOHGB0v0NPZwYXnJrPoPRefy/ahXrYN9bH9nD62V/3ppQ4zazSH+2laXg6+8fdH2PvMC3zp/71EcWmZqy4+l/90w0/yq3suYqDXQ2xmZ5+TZ4MWSkt84ms/ZO83D3HwyDHO2dTNTT9zCf/8jRdz5YVD67+AmdkZ5HDfgNHxGe787P/luy9O86bLtvCBX97Ftbu309fti5dm1hoc7qcgIvjk13/IvV98noHeLh68dYRfunJbs8syMzuJwz2liZl5/u2fPMffjE7yi68Z5qPv2MPwYG+zyzIzq8vhnsJffWec3/uz55hdKPHh63bz7jddelo9PszMzjSH+xqOFUt8+C+/y95vHuJ1Fw7x3991FVdcMNjssszM1uVwX8XM/CLv/Nj/5nvjM7zvzZfxgbfuorfLF0zNrD043OtYXg5+5/P7GZso8Me3vpFffO0FzS7JzOyU+K2RdfzRV77Pl78zzu+//UoHu5m1JYd7jSe/N8F//atRrr/qIn7z6h3NLsfMbEMc7lV+eHiWO/d+iyu3D3HvDT/lHTFm1rYc7mWzCyXe9/CzdHSIj9/yBrfKNbO25nAneefp7/7Zc4xNzPCH7/pHXLxlc7NLMjM7LQ534IGv/j2PP/civ/srr+WancPNLsfM7LTlPtyfHjvMR554nn/2kxfyvjdf1uxyzMwaItfh/sLRY7x/79+y84JBPvoOX0A1s+zIdbj/x8e/w9Jy8PFb3kC/b6phZhmS63B/7kdTvOXKbezY2t/sUszMGiq34T49v8iLU/NcccFAs0sxM2u43Ib72HgBgF3b3OXRzLInx+E+A8CubZ65m1n2pAp3SVskPSppVtJBSTetct65kj4paaL8cXdDq22g0fECfd0dXHye37BkZtmTdovIfUAR2AZcBTwuaX9EHKg57w+AzcAO4ALgf0k6GBGfaFC9DTM2McMVFwzQ0eHtj2aWPevO3CX1AzcCd0VEISKeBh4Dbqlz+q8BH42IYxHxQ+BB4DcbWG/DjI0X2OW7KplZRqVZltkFLEXEaNWx/cDuVc5Xzeev32BtZ8zU3CIvTc+z0xdTzSyj0oT7ADBVc2wKqJeMTwAflDQo6QqSWXvdRW1Jt0naJ2nf5OTkqdR82r4/kVxM3eltkGaWUWnCvQAM1RwbAmbqnPvbwBwwBvwFsBf4Ub0XjYgHImIkIkaGh89us65Rb4M0s4xLE+6jQJeknVXH9gC1F1OJiKMRcXNEbI+I3eXX/2ZjSm2c0fEZNnV38urzNjW7FDOzM2Ld3TIRMSvpEeAeSe8l2S1zHfCztedKuhx4pfzxVuA24OcbWnEDjI0XvFPGzDIt7ZuY7gA2ARMkSy23R8QBSddIKlSd9wbg2yRLNvcCN9fZLtl0YxMz7PSbl8wsw1Ltc4+Io8D1dY4/RXLBtfL488DnG1bdGTA1t8j49ILX280s03LXfsBtB8wsD3IX7pWdMjv9BiYzy7AchnuyU+ZV53qnjJllV+7CvXIx1TtlzCzL8hfu4wUvyZhZ5uUq3KeOLTIxs+CLqWaWebkK99GJyk4Zz9zNLNvyFe7lbZB+A5OZZV2uwn1svEB/j3fKmFn25SrcR8eTuy9J3iljZtmWs3Av+AYdZpYLuQn3l2eLHC54p4yZ5UNuwn1sotx2wDN3M8uB3IT76Li3QZpZfuQm3MfGZxjo7eKic/qaXYqZ2RmXm3AfLd99yTtlzCwPchPuYxMzvphqZrmRi3A/OlvkcKHohmFmlhu5CPcxtx0ws5zJRbiPlrdBeqeMmeVFLsJ9bHyGwd4uLvROGTPLiVyE++j4DFds804ZM8uPXIT72HiBXb6YamY5kvlwP1JY4Mhs0RdTzSxXMh/u7iljZnmU/XBf6SnjmbuZ5Ufmw310vMBgbxfbh7xTxszyIwfhPsNO75Qxs5zJfLiPTRT85iUzy51Mh/vhwgJHZ4u+mGpmuZPpcB8br7Qd8MVUM8uXbIf7hO++ZGb5lCrcJW2R9KikWUkHJd20ynm9kj4maVzSUUlfkPSqxpac3qEjx+jr7uCCwd5mlWBm1hRpZ+73AUVgG3AzcL+k3XXOuxP4J8BPARcBrwB/2IA6N2R6fpFzN/V4p4yZ5c664S6pH7gRuCsiChHxNPAYcEud038C+FJEjEfEPPBZoN7/BM6K6bkSQ5u6mvXtzcyaJs3MfRewFBGjVcf2Uz+0HwSulnSRpM0ks/wv1ntRSbdJ2idp3+Tk5KnWncr0/CJDfd1n5LXNzFpZmnAfAKZqjk0B9a5SjgKHgH8ApoErgXvqvWhEPBARIxExMjw8nL7iUzA9v8jQJoe7meVPmnAvAEM1x4aAmTrn3g/0AecD/cAjrDJzPxum50oM9XlZxszyJ024jwJdknZWHdsDHKhz7h7goYg4GhELJBdTf1rS1tMv9dR55m5mebVuuEfELMkM/B5J/ZKuBq4DHq5z+jPAeySdI6kbuAP4cUQcbmTRaUQE03NeczezfEq7FfIOYBMwAewFbo+IA5KukVSoOu/fAPPAGDAJvB349QbWm9pscYnlwLtlzCyXUiVfRBwFrq9z/CmSC66Vx0dIdsg03fTcIoBn7maWS5ltPzA9Xw53r7mbWQ5lN9znSoBn7maWTxkO98rM3WvuZpY/2Q33ea+5m1l+ZTfc57zmbmb5ld1wn0/W3Af9DlUzy6HshvvcIpt7OunuzOx/opnZqjKbfO4IaWZ5lt1wdy93M8ux7Ia7Z+5mlmPZDnfvlDGznMpuuLuXu5nlWHbD3TN3M8uxTIa7e7mbWd5lMtzdy93M8i6T4e5e7maWd9kMd/dyN7Ocy2a4u5e7meVcRsPdvdzNLN+yGe7u5W5mOZfNcHcvdzPLuWyGu3u5m1nOZTPc3cvdzHIuk+nnjpBmlneZDPepuUXvlDGzXMtkuCcdIT1zN7P8yma4uyOkmeVcdsPdO2XMLMeyGe5zJc/czSzXMhfuy8vBjHfLmFnOZS7cZ4sl93I3s9xLFe6Stkh6VNKspIOSblrlvC9KKlR9FCV9u7Elr63y7lTP3M0sz9JOb+8DisA24CrgcUn7I+JA9UkR8bbqx5K+Avx1A+pMzX1lzMxSzNwl9QM3AndFRCEingYeA25Z5+t2ANcAD59+men5LkxmZumWZXYBSxExWnVsP7B7na97D/BURPxgo8VtxMqyjNfczSzH0oT7ADBVc2wKGFzn694DPLTak5Juk7RP0r7JyckUZaRTmbmf42UZM8uxNOFeAIZqjg0BM6t9gaSfA7YDf7raORHxQESMRMTI8PBwmlpT8Y06zMzShfso0CVpZ9WxPcCBVc4HuBV4JCIKp1PcRlTun+pe7maWZ+uGe0TMAo8A90jql3Q1cB2rXCiVtAl4J2ssyZxJ0/OL9Pd00uVe7maWY2kT8A5gEzAB7AVuj4gDkq6RVDs7v55kTf7JxpWZ3vScm4aZmaVau4iIoyShXXv8KZILrtXH9pL8D6ApfKMOM7MMth9ImoZ5vd3M8i174e6Zu5lZRsPda+5mlnPZC/e5km/UYWa5l6lwX+nl7pm7meVcpsJ9pZe719zNLOcyFe5uGmZmlshWuLvdr5kZkNVw95q7meVctsLdt9gzMwOyFu4rM3evuZtZvmUr3N3L3cwMyFq4u5e7mRmQtXB3L3czMyBr4e5e7mZmQNbC3R0hzcyArIW7e7mbmQFZC3fP3M3MgCyGu9fczcwyFu7u5W5mBmQo3N3L3czsuMyEu3u5m5kdl5lwdy93M7PjshPu7uVuZrYie+HuNXczswyFu3u5m5mtyEy4T7mXu5nZisyEu9fczcyOy064l2/U4V7uZmZZCve5knu5m5mVZSYJ3VfGzOy47IT7nDtCmplVpAp3SVskPSppVtJBSTetce4/lvRVSQVJ45LubFy5q0tm7l5vNzOD9DP3+4AisA24Gbhf0u7akyRtBZ4APg6cD1wBfLkxpa4t6QjpmbuZGaQId0n9wI3AXRFRiIingceAW+qc/gHgSxHxmYhYiIiZiPhuY0uuz2vuZmbHpZm57wKWImK06th+4KSZO/Am4Kikr0uakPQFSZc0otD1JGvuXpYxM4N04T4ATNUcmwIG65z7auBW4E7gEuAHwN56LyrpNkn7JO2bnJxMX3Edy8vBzELJM3czs7I04V4AhmqODQEzdc6dAx6NiGciYh74D8DPSjqn9sSIeCAiRiJiZHh4+FTrPrHAYolwL3czsxVpwn0U6JK0s+rYHuBAnXOfA6LqceVzbay8dKbdV8bM7ATrhntEzAKPAPdI6pd0NXAd8HCd0z8B/LqkqyR1A3cBT0fEK40sutb0nDtCmplVS7sV8g5gEzBBsoZ+e0QckHSNpELlpIj4a+D3gcfL514BrLonvlEqfWW85m5mlki1jhERR4Hr6xx/iuSCa/Wx+4H7G1JdSu4IaWZ2oky0H/D9U83MTpSNcPfM3czsBNkId/dyNzM7QTbCfa7EQG+Xe7mbmZVlIg2n5916wMysWjbCfc5Nw8zMqmUj3Od9ow4zs2rZCPe5krdBmplVyUa4e+ZuZnaCbIS719zNzE7Q9uG+0svdu2XMzFa0fbiv9HL3zN3MbEXbh7tbD5iZnSwD4e6mYWZmtdo/3Oc9czczq9X+4T7nG3WYmdVq/3Cf9y32zMxqtX+4++bYZmYnaf9wL6+5D/Q63M3MKto/3N3L3czsJG2fiO7lbmZ2svYPd/eVMTM7SfuHuztCmpmdpP3D3b3czcxO0v7h7pm7mdlJ2j/cveZuZnaStg5393I3M6uvrcN9ZsG93M3M6mnrcHcvdzOz+to73OfdV8bMrJ72Dvc5d4Q0M6unvcN93r3czczqSRXukrZIelTSrKSDkm5a5by7JS1KKlR9XNbYko87v7+Ht71+O8ODvWfqW5iZtaW0i9X3AUVgG3AV8Lik/RFxoM65n4uIdzeqwLWM7NjCyI4tZ+NbmZm1lXVn7pL6gRuBuyKiEBFPA48Bt5zp4szMbGPSLMvsApYiYrTq2H5g9yrn/5qko5IOSLp9tReVdJukfZL2TU5OnkLJZma2njThPgBM1RybAgbrnPt54EpgGPgt4N9Jele9F42IByJiJCJGhoeHT6FkMzNbT5pwLwBDNceGgJnaEyPiOxHx44hYioivA/8NeMfpl2lmZqciTbiPAl2SdlYd2wPUu5haKwBtpDAzM9u4dcM9ImaBR4B7JPVLuhq4Dni49lxJ10k6T4mfBn4b+ItGF21mZmtL+yamO4BNwASwF7g9Ig5IukZSoeq83wC+T7Jk8yngIxHxyUYWbGZm60u1zz0ijgLX1zn+FMkF18rjuhdPzczs7FJENLsGJE0CBzf45VuBww0sp5Fc28a0cm3Q2vW5to1p19oujYi62w1bItxPh6R9ETHS7DrqcW0b08q1QWvX59o2Jou1tXXjMDMzq8/hbmaWQVkI9weaXcAaXNvGtHJt0Nr1ubaNyVxtbb/mbmZmJ8vCzN3MzGo43M3MMqhtwz3t3aGaRdJXJM1X3ZHqe02q4/3l1soLkh6qee6XJD0v6ZikJyVd2gq1SdohKWru6HXXWa6tV9KD5Z+tGUnfkvS2quebNnZr1dYiY/dpSS9KmpY0Kum9Vc81+2eubm2tMG5VNe4sZ8enq47dVP77npX055LWv0tRRLTlB0kbhM+RvEP250jaEO9udl1V9X0FeG8L1HEDybuL7wceqjq+tTxm7wT6gP8MfKNFattB0nSuq4nj1g/cXa6lA/hVkrYaO5o9duvU1gpjtxvoLX/+WuAl4A3NHrd1amv6uFXV+GXgKeDTVTXPAG8u593/AD673uukvc1eS6m6O9TrI6IAPC2pcneoDza1uBYTEY8ASBoBXl311A3AgYj4k/LzdwOHJb02Ip5vcm1NF0nDvLurDv2lpB+QBMH5NHHs1qnt2TP9/dcTJ95+M8ofl5PU1+yfudVqO3I2vv96JP0G8ArwdeCK8uGbgS9ExFfL59wFfFfSYESc1Hq9ol2XZU717lDNcq+kw5K+JukXml1Mjd0kYwasBMbf0VpjeFDSjyR9QtLWZhYiaRvJz90BWmzsamqraOrYSfojSceA54EXgf9Ji4zbKrVVNG3cJA0B9wC/U/NU7bj9Hck9rXet9XrtGu6ncneoZvk94DLgVST7VL8g6fLmlnSCVh7Dw8AbgUtJZnuDwGeaVYyk7vL3/2R5htkyY1entpYYu4i4o/y9ryFpGb5Ai4zbKrW1wrh9GHgwIl6oOb6hcWvXcE99d6hmiYj/ExEzEbEQSdvjrwFvb3ZdVVp2DCO5Efu+iChFxDjwfuCt5ZnNWSWpg+TeBcVyHdAiY1evtlYau0juyPY0yZLb7bTIuNWrrdnjJukq4C3AH9R5ekPj1pZr7lTdHSoixsrH0t4dqlla7a5UB4BbKw/K1zEupzXHsPJOu7M6fpIEPAhsA94eEYvlp5o+dmvUVqspY1eji+Pj02o/c5Xaap3tcfsFkou6h5K/WgaATkmvA54gybekIOkyoJckB1fX7CvDp3FF+bMkO2b6gatpod0ywLnAtSQ7ArpILojMAq9pQi1d5TruJZnlVWoaLo/ZjeVjH+Hs71xYrbafAV5D8pvl+SS7op5swth9DPgGMFBzvBXGbrXamjp2wAUkN+0ZADrL/w5mSe7e1tRxW6e2Zo/bZmB71cd/Af60PGa7gWmSZaR+4NOk2C1z1n4Yz8BgbAH+vPyXcwi4qdk1VdU2DDxD8mvTK+V/hL/cpFru5viugMrH3eXn3kJyUWmOZOvmjlaoDXgX8IPy3+2LJHf12n6Wa7u0XM88ya/FlY+bmz12a9XW7LEr/+z/Tfnnfhr4NvBbVc83c9xWra3Z41an1rspb4UsP76pnHOzJLcu3bLea7i3jJlZBrXrBVUzM1uDw93MLIMc7mZmGeRwNzPLIIe7mVkGOdzNzDLI4W5mlkEOdzOzDHK4m5ll0P8HcyDWzinP0RMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(L(learn.recorder.values).itemgot(2));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.982826292514801"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.recorder.values[-1][2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Going Deeper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.082089</td>\n",
       "      <td>0.009578</td>\n",
       "      <td>0.997056</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dls = ImageDataLoaders.from_folder(path)\n",
    "learn = cnn_learner(dls, resnet18, pretrained=False,\n",
    "                    loss_func=F.cross_entropy, metrics=accuracy)\n",
    "learn.fit_one_cycle(1, 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Jargon Recap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Questionnaire"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. How is a grayscale image represented on a computer? How about a color image?\n",
    "1. How are the files and folders in the `MNIST_SAMPLE` dataset structured? Why?\n",
    "1. Explain how the \"pixel similarity\" approach to classifying digits works.\n",
    "1. What is a list comprehension? Create one now that selects odd numbers from a list and doubles them.\n",
    "1. What is a \"rank-3 tensor\"?\n",
    "1. What is the difference between tensor rank and shape? How do you get the rank from the shape?\n",
    "1. What are RMSE and L1 norm?\n",
    "1. How can you apply a calculation on thousands of numbers at once, many thousands of times faster than a Python loop?\n",
    "1. Create a 3×3 tensor or array containing the numbers from 1 to 9. Double it. Select the bottom-right four numbers.\n",
    "1. What is broadcasting?\n",
    "1. Are metrics generally calculated using the training set, or the validation set? Why?\n",
    "1. What is SGD?\n",
    "1. Why does SGD use mini-batches?\n",
    "1. What are the seven steps in SGD for machine learning?\n",
    "1. How do we initialize the weights in a model?\n",
    "1. What is \"loss\"?\n",
    "1. Why can't we always use a high learning rate?\n",
    "1. What is a \"gradient\"?\n",
    "1. Do you need to know how to calculate gradients yourself?\n",
    "1. Why can't we use accuracy as a loss function?\n",
    "1. Draw the sigmoid function. What is special about its shape?\n",
    "1. What is the difference between a loss function and a metric?\n",
    "1. What is the function to calculate new weights using a learning rate?\n",
    "1. What does the `DataLoader` class do?\n",
    "1. Write pseudocode showing the basic steps taken in each epoch for SGD.\n",
    "1. Create a function that, if passed two arguments `[1,2,3,4]` and `'abcd'`, returns `[(1, 'a'), (2, 'b'), (3, 'c'), (4, 'd')]`. What is special about that output data structure?\n",
    "1. What does `view` do in PyTorch?\n",
    "1. What are the \"bias\" parameters in a neural network? Why do we need them?\n",
    "1. What does the `@` operator do in Python?\n",
    "1. What does the `backward` method do?\n",
    "1. Why do we have to zero the gradients?\n",
    "1. What information do we have to pass to `Learner`?\n",
    "1. Show Python or pseudocode for the basic steps of a training loop.\n",
    "1. What is \"ReLU\"? Draw a plot of it for values from `-2` to `+2`.\n",
    "1. What is an \"activation function\"?\n",
    "1. What's the difference between `F.relu` and `nn.ReLU`?\n",
    "1. The universal approximation theorem shows that any function can be approximated as closely as needed using just one nonlinearity. So why do we normally use more?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further Research"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Create your own implementation of `Learner` from scratch, based on the training loop shown in this chapter.\n",
    "1. Complete all the steps in this chapter using the full MNIST datasets (that is, for all digits, not just 3s and 7s). This is a significant project and will take you quite a bit of time to complete! You'll need to do some of your own research to figure out how to overcome some obstacles you'll meet on the way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "jupytext": {
   "split_at_heading": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
